Exploring Datascript with a small application

Overview

In a previous post I have briefly described how using ClojureScript, Rum and Datascript helped me to explore a new way to prototype software starting by investigating how the user is going to interact with it.

People, instead, showed more interest on Datascript (go figure :) ) so in this post I am going to write a small application to explore how to use Datascript and Datalog without digging too much into the nitty gritty details of both because, I think, there are better resources out there. (See Links section.)

The application

Instead of writing the usual TODO app, I would like to provide a way for users to give feedback to my blog posts. Usually this is why you add a comment area to your website but it would be nice if readers could write their feedback for each section and not just the whole post.

For now I want to focus on the frontend part of it, to get a feel of how it could work in the hands of the users. If it proves to be usable I can think of a way to store the feedback to a durable storage, eventually add access control, eventually an admin interface etc etc.

To keep things super simple I will target the structure of my blog posts which is more or less like this:

<div class='body'>
  <div class='section' id='section-title'>
    <h2>Section title</h2>
    <p>Text<p/>
  </div>
</div>

The div with class body is the main container; inside there are N section div elements, each with an id which is the slugified title of the section, an h2 element with the title of the section and finally the content.

At start the application will fetch all post's sections, will store their id and title in the database, and will attach a react component to them which will: - provide a "Write your feedback" button which will toggle the feedback area composed by - a form to write new comments - a list of old comments with buttons to reply to or to delete them

The sources of the application are available here.

I suggest to try the full application to get a feel of how it looks and works; to run it please follow the instructions in the README.md of the project.

Stack for the prototype

To build this application I have chosen to use my (currently) preferred prototyping stack which is composed by:

  • ClojureScript: the language

  • Figwheel.main: build system and live code update

  • Rum: React wrapper library

  • Datascript: in memory Datalog database

For me, this work pretty well and I feel quite productive with it; the main benefit is the fast feedback loop provided by figwheel, the easy of use of Rum, a powerful, easy to use, database where to store the application state and last but not least a REPL where I can quickly fiddle with every bit of the application.

Data model

For this simple application there are only two entities that we are going to store in the database: section and comment; lets define a data model that will hold the application state:

section

  • id: string, HTML id of the section

  • name: string, name/title of the section as extracted from the HTML

feedback/comment

  • id: uuid

  • author: string, author of the comment

  • text: string, the content itself

  • created-at: string, ISO format of the time of the creation

  • parent: optional, ref to parent comment

The data model can be described with the following schema:

(def schema
  {:section/id      {:db/unique :db.unique/identity}
   :comment/id      {:db/unique :db.unique/identity}
   :comment/section {:db/valueType :db.type/ref
                     :db/cardinality :db.cardinality/one}
   :comment/parent  {:db/valueType :db.type/ref
                     :db/cardinality :db.cardinality/one}})

As you may have noticed, not all attributes of the entities are in the schema, if fact it is sufficient to describe attributes with special meaning like ids or references.

Data layer

Usually I like to have a separate namespace for the data and presentation layers, even for applications as simple as this one. Lets have a look at the app.db namespace.

After defining the schema, the db gets created with the following form:

(def db (d/create-conn schema))

It is common to use defonce when defining app state to not reset it after a code reload but, in this case, I prefer to def because, when experimenting, it is quite useful to be able to change the schema and you may want to use the newest one in your database.

Another important tool for experimentation is to quickly populate the database with mock data; I find it very useful when it comes the time to work on the UI or when I need to test queries and transactions.

Here is an example query and the result when some data is in the database:

(defn all-sections [db]
  (d/q '[:find ?id ?name
         :where
         [?s :section/id ?id]
         [?s :section/name ?name]
         ]
    db))

(comment
  (all-sections @db) ;; => #{["overview" "Overview"] ["application-design" "Application design"]}
  )

Digging a bit more in to the db namespace, we can find an example of how using aggregates looks like in the section-comments-count function, here is the code:

(defn section-comments-count
  "If the query cannot find any entity that matches then it cannot reasonably
   count anything and it will return nil; with the use of `or` we can handle
   this case and return 0."
  [db section-id]
  (or
    (d/q '[:find (count ?c) .
             :in $ ?section-id
             :where
             [?s :section/id ?section-id]
             [?c :comment/section ?s]
             ]
        db section-id)
    0))

As stated in the docstring, when there are no matching entities, the query will return nil, for this reason I use an or to return the correct value (zero).

Also, please note the dot after the call to count, this signals that we want a scalar value back instead of a set with one element containing the count.

Building the UI

After a quick tour of the data layer is now the turn of the UI part which can be found in the app.core namespace. This namespace make more sense if read bottom to top while the db namespace is better read top to bottom.

At start, the application looks for all section elements, stores the section id and name of the section in the detabase and attaches a feedback component to it. This can be seen it the following code fragment:

(defn init-app
  "fetches all elements with a section class, parse all sections,
   remove prevoius feedback related elements (useful when reloading script)
   and add them to the db, finally attach a `feedback` button to all
   sections' container div"
  [db]
  (let [sections (dom/getElementsByClass "section")]
    (doseq [section sections]
      (attach-feedback-component-to-section section db))
    (start-listener!))) ;; start-listener will be addressed later

Each component will receive the database instance and will react to changes; this is possible because a database is basically an atom so using rum/reactive mixin works out of the box, here is the main component's code:

(rum/defc feedback-component < rum/reactive
  "feedback area is composed by an input text for the author,
   a textarea for the message and a div containing all previous messages
   for the current section identified by `section-id`"
  [section-id db]
  (let [db (rum/react db) ;; <- the component will react to changes of the db
        area-id (str "feedback-area-" section-id)
        comments (sorted (db/section-comments db section-id))
        comment-count (db/section-comments-count db section-id)]
    ;; view code omitted for brevity
    ))

The following binding does most of the magic:

db (rum/react db)

Simplifying, it subscribes to the db atom and derefs it, like we would do with @db or (deref db); a this point it will be possible to use the returned reference in queries like:

comments (sorted (db/section-comments db section-id))

Closing words

The rest of the application details are out of scope and it should be trivial to figure them out by reading the documentation of libraries used to build the application.

I hope that this quick tour of a simple application may have given you some pointers to start to work with Datascript and eventually Rum.

Links section

My new approach to software prototyping and the stack that makes it possible

Foreword

In this blog post I am going to illustrate how, during the development of a pet project, I have changed the way I approach software development and prototyping, going from "backend first" to "frontend first, backend eventually". I will also mention the tools that make it possible and even enjoyable for a non frontend developer.

Background

At $work I have always spent most of my time in the backend and infrastructure side of things but sometimes, especially when working on side projects, I have to go "full stack" and take care of frontend tasks, even if I kind of suck at it. For this reason I am always inclined to start with some sort of backend code, starting with a data model, core business logic and a REST API to (eventually) build a frontend on top of it...which most of the times means I will lose interest/momentum and the project dies.

At some point something clicked

One day (probably a lazy Sunday) I had an idea: I want an incredibly stupid way to collect random thoughts, links or any kind of text blob, I want to own it, it must live on my VPS and I have to finish it today! This time thoughts decided to follow a different path:

  • how important what I am going to write here is? can I afford to lose data?

  • what if I don't need to store anything in a durable storage?

  • ok, aim for least amount of features and see later if I need more

The outcome is a service that will keep all data in volatile memory, losing it at each restart and this is perfectly fine for what I needed at that time! I think this small tool planted the seed for what, I think, will be be my preferred method for future projects.

What has been so different with development of this tool is that I have tried and (more or less succeeded) to address the problem at hand without much ceremony, and it served its purpose quite well. At some point I've started to add more features "just because" but, at the end of the day, the core remained the same and now I even want to remove these new features because they are not adding value but are, effectively, distracting.

Next project

Fast forward few months, I am again thinking about a new project. My girlfriend needed/wanted a tool to make it easy to share a watermarked version of her drawings, even better if the shared link required a password to access the image.

Compared to the previous project, this one is slightly more complex, especially the user facing part, but I am confident that if I address the most basic requirements first, I should be able to give her a rough beta in few hours. Here are the features I have decided to include in the "MVP":

  • user can upload a file

  • user can chose a watermark size

  • user can eventually chose a password to protect the image

  • once the user clicks "Upload", the system applies the watermark and

  • render a page with the list of uploaded/watermarked files, each with a link to it

So far so good, the tool is there, ugly but working. What if we open this tool to other people? WOW! We can make our own product! Lets do it!

Armed with great energy I have started working on the new version but this time I have started building a full featured backend even if, at this point, is not 100% clear how a user is going to interact with it, how valuable it could be and all the usual unknowns. So why I am insisting with the old approach when it showed, multiple times, it is quite ineffective? I guess it is a matter of habits (because I don't want to call it stupidity).

A new approach

I often fail to make progress on my projects when it comes the turn of the frontend code, after having written most of the backend, maybe this time I can try the other way around! Common sense suggests to create mocks of the interface using tools like Balsamiq, Moqups and so on but, for some reasons I am totally unable to use those kind of tools plus I would like to have a taste of how to UI feels in the hands a user...what should I do now?

Recently I have started experimenting with ClojureScript and Rum (a library that wraps React) and so far I feel quite productive with it, why should I not use these tools to write the mock? The plan now looks like this:

  • quickly build an ugly but functional UI

  • mock the app state/data to reflect what I need in the UI

  • understand what feels wrong, get feedback from other people

  • plan next iteration

  • repeat until happy

Essential to this approach is the quick feedback loop you can get by using ClojureScript + Rum (or any other React wrapper like Reagent), immutable persistent data structures and a REPL; I "suspect" that, if we exclude the REPL, a similar setup can be achieved with other stacks but I am quite happy with the tools I have chosen.

How has it been so far?

I am not a frontend developer and every time I try to create some sort of UI I am pretty sure I will fail, lose interest and put the project on hold but, this time, it has been quite different. Being able to iterate quickly and having a fast feedback loop, helped me a lot to proceed at "great speed" and not losing focus. Each time I wanted to add a new feature or change how a component had to work, the process has been pretty straightforward:

  • create a new branch

  • try out the new idea by small, even tiny iterations with live feedback

  • adjust the data model as needed

  • repeat

  • if happy merge to master otherwise scrap everything and forget

Managing the app state

One of the points that I would like to spend so time on is the app state. Even if keeping all the state in one big map is quite handy and flexible, at some point, it becomes harder and harder to maintain, especially in the case where there are many ( > 4 or 5) entity types with cross references and the size of the data gets bigger and bigger; it can still be done but the state management code starts to get bigger and harder to maintain compared to the UI code; for my case, the UI code had to know too much about the data model itself impacting the separation of concerns and this was driving away the focus on the main goal which is to explore the space of possibilities for the UI I want to build.

Introducing Datascript into my tool-set

With the growing pain of state management I have decided to invest a little bit of time to find out which solutions are already available in this space and fortunately I have stepped into Datascript, an in memory database developed by the same person who gave us Rum :) This database has all the features I need plus some more; I am sure that I am still missing some handy features but so far I have been able to:

  • build a consistent data store

  • which supports cross references between entities

  • which I can easily query (using Datalog)

  • with support to joins

Datalog itself is quite simple even if a bit "alien" until you know how to use it; instead of writing about it here I prefer to give a link to the best learning material I have found http://www.learndatalogtoday.org/.

Armed with Datascript (for the data store) and Datalog (for the queries) the UI code can now interact with the app state without knowing much about it! This is a great relief because now I can focus again on building a UI for my project and spend the right amount of time (< 5%) on the data store.

Another good point of this approach is that, when the backend will be ready, it will be a matter of translating what I will get out of its APIs to the UI's internal data store and everything will continue to work as expected.

Summary

The goal of this post was to write down my current approach to software development with the hope to start a discussion with other developers which may be interested to explore and talk about novel methodologies which have worked great for them. If by reading this blob of text you are now also curious about ClojureScript and its ecosystem, I am even more happy!

Routing Celery task for simple prioritization

Problem

Like most businesses, where I work we need to send lots of notifications to our users, mainly emails and push notifications; the setup is quite simple:

  • a service accepts requests to send notifications to users

  • the notification service prepare the message and put it in a queue

  • a pool of workers fetches messages from the queue and perform the actual delivery

This works reasonably well and we can scale the service increasing the instances of the notification service and the delivery workers.

This setup is used also when a user requests an export of her/his historical data; since this process can take a while, a background job fetches the data, generates a pdf and sends it via email using the notifications service. At the same time we generate hundreds of thousands of notifications, usually in the morning, and this fill up the notifiation queue so if a user requests an export during this time frame its email will have to wait a lot before it can be processed.

Solution

We have evaluated a couple of solutions:
  • per message priority

  • dedicated low priority queue for high volume automaticly generated notifications using task routing

The first solution is a generic one but as far as we have seen it is not easy to have the guarantie that a high priority message will be delivered in our desired time frame and we opted for the second solution because we don't need a fine grained prioritization system (maybe in the future) but just a way to continue to deliver user generated notifications when we are sending our automated high volume notifications during the morning.

Implementation

Our stack is based on Celery and it is composed mainly by two parts:
  • the notifications service thats send messages to a queue

  • a pool of workers that fetch messages from the queue and deliver the notifications

To achieve our goal we only had to change the way that notifications service sends messages to the queue by specifing the low or default priority queue based on the message type and by running a specific pool of workers bound to each priority queue.

Example code with routing:

from celery import Celery

celery = Celery()
celery.main = 'Test app'
celery.config_from_object('celeryconfig')
celery.autodiscover_tasks(
    ['tasks'],
    force=True
)


@celery.task
def task(prio, message):
    print(f'{prio}: {message}')

# calling the task specifying the queue
task.apply_async(args=('default', next(generator)),
                 queue='default')
How to run a worker specifing the queue::

$ celery -A tasks worker --loglevel=INFO -Q default

This solution works but there is an efficency problem since when the low priority queue will be empty the low priority workers will be idle wasting precious (and paid) resources; fortunately there is a simple solution for this because it is possible to specify more than one queue using the -Q parameter. This way we will have a dedicated pool of workers that will work on messages generated by user activity and a second pool of workers that will handle the automated messages and, when those will be finished, these workers can help with default priority messages.

An example implementation is provided in this repo with instruction to run the different use cases.

P.S. We are hiring!

Evaluating Rust for game development

Evaluating Rust for game development

There is no doubt that Rust is gaining popularity as a system development language and most of the time games and game engines are built using languages such as c or c++ which were born with the same target in mind, being able to access all the capabilities that a machine has to offer with the minimum runtime overhead.

In the last years, powerful engines and tools have offered a pletora of bindings for different languages like C#, Lua, Python, Javascript and so on with great results; for example Unity have had and continue to have a great success providing a powerful platform to build game on so why taking the risk and effort to use such a low level language to build a game?

As an "old school" developer I prefer to be in control of my software development stack and often, instead of creating the game that I have in mind, I start creating an engine to build my game on...but why? That's because it is fun! Or because for some simple game it seems to be easier to create a small library (or extend some old code) instead of using a powerfull solution like Unreal Engine, or and or and or...

I have started creating games again and again using high level languages like Python, Javascript, Lua and even the good old c++; in general I really like to create prototypes with python but it is not easy to distribute your games and almost impossible (or reaay hard) to target mobile platforms. Another option is to target the browser with the growing set of available engines but again, targeting mobile platforms can be a problem and targeting consoles is almost impossible.

Rust on the other end is a very young player in the field but with its ability to target many architectures I think that starting to think about using it can be a safe bet; there is at least one success story for using rust for game development and the final result is encouraging<https://www.rust-lang.org/pdfs/Rust-Chucklefish-Whitepaper.pdf>.

List of available resources

Right now the best collection of rust game development related project can be found in the "Are we game yet" page<http://arewegameyet.com/>. this web site maintain a list of frameworks and engines fro the lowest level flavour to the most complete game engine.

Of all the available solutions ggez<https://github.com/ggez/ggez/> (Good Game Easy) is the one which attracts me the most because I like to create small prototypes and not having the knowledge overhead required by other full featured engines or frameworks like piston<https://github.com/PistonDevelopers/piston> or amethyst<https://github.com/amethyst/amethyst> it is helpful when experimenting; it also feels like a microframework to which you can add the components you need without having to comply with other people choices and taste.

Cython and C pointers

Cython is a powerful tool that can speed up your Python code or can help you to quickly create an extension module for a library that has c bindings.

This post is not a cython tutorial but its about a problem I have encountered during the development of a wrapper to Wakaama LWM2M library; if you want to learn more about Cython please refer to its documentation.

During the development of said wrapper I have had a nasty strange bug affecting my code, at some point some object referenced by a pointer kept at the C library side seemed to change its value! For example think about this situation:

  • you pass an object to a certain c function

  • the object's pointer is stored internally by the c library

  • you retrieve the object from the c library but its content its not the same...it's not even an instance of the same class!

To reproduce this behaviour please have a look at overwrite.py in my test project.

At first I've been thinking about memory corruption caused by the C library code; this wrong assumption costed me at least two days of messing with gdb (that saved my life in another memory related bug) that have not given any useful insight.

I don't remember how I've spotted this problem but probably I should have tryed everything until something pointed me to the right direction.

What seems to happen here is something like this:

  1. create an object and assign it to a variable A

  2. store its pointer in the C library

  3. get back the pointer of the object; it will be the same object as referenced by A

  4. assign a new object to A

  5. create an object assign it to a variable B

  6. get back the pointer of the object; it will NOT be the same object as referenced by A but it will be the one referenced by B

When the variable A is referencing the new object the old one is not referenced by anyone in the python world and its memory will became available to store the new object assigned to B and the pointer in the C world is now pointing to what is referenced by B.

It may seems obvious but when it happened to me it was inside a callback originating in the C code and being handled by a function defined in the cython context.

Lesson learned

When working with two different languages together nothing is naive especially when you are wrapping some code that you don't really know and that can bite your ass as soon as you think "what can go wrong?"

The joy of contributing to open source/free software

From time to time I like to give back to the community with small patches or trying to solve issues, sometimes my contribution comes from bugs found while implementing some kind of software and sometimes I just notice something that can be fixed or improved.

Usually to process is a kind of cold syntetic handshake between my code and whatever (D)VCS system used to accept my work but it happens that you will find some real person at the other side and this is especially true at Mozilla.

In my opinion Mozilla infrastructure and people are key factors in welcoming new developers; there are clear documents about the processes which you will be pointed to by the team and there are real persons that will help you until you master the procedure.

Which are the take aways of contributing?

First of all you are forced to read someone else's code; if you work alone or your employer does not force code reviews this is really helpful because you are probably going to learn a lot from other sources.

Similarly when contributing to other people's code you are forced to add unit tests to your patches otherwise your code will not be accepted; this is ideal in the situations where in your day to day work tests are seens as useless nerd goodies.

When contributing to bigger projects there is usually a strong procedure that must be followed; this is not just bureaucracy but it must be seen as a backbone of the development/deployment/release process. Knowing that any regression "shall not pass" is a huge boost to your confidence while creating your patch.

And finally people! I think that is the most amazing part of the work; you'll get to know persons that will teach you something (or maybe the opposite), you will have to communicate effectively to make your point or just to understand what problem you are going to fix, increasing your communication skills; oh and don't we forget the joy of talking to nice people :)

Conclusions

Contributing can be a beautiful and useful experience for beginners, intermediate and senior developers, everyone can learn something while doing useful work for the community (which in turn will provide a better tools for yourself too); even if the amount of free time you have at hand is a couple of hours a week I suggest trying at least to document some software you use or fix that annoying bug that is driving you crazy, you will thank yourself!

P.S.

I'd like to thank Andrew Halberstadt for his help and his patience while working on some issues on Mozilla Central repo/bugzilla, thank you Andrew!

cx_Freeze, again

We meet again, xz_Freeze

It's been a long time since the last post and for sure I was not thiking about another post about cx_Freeze. I have spent this week end trying (and at the end succeding) to build an update to an old software. For future memory I'll write briefly the steps needed to get a functional build without losing my mind.

Brief description of the problem

The software is a desktop application used to upload email attachments to a web based data store; the application talk to the web app using a "RESTy" api, and fetches the file from a email address; the GUI is written with the beatiful PyQT5 (someone may prefere PySide) and usually this dependency is a pain to work with but I have to admit that the newer version installs gracefuly.

OS version and things to download

The person using this software needs to run it from a Windows OS and at this time Windows 7 is the oldes worsion of the OS I want to support (no Vista, please) since addressing Windows 10 will produce an executable with no retro compatibily and Windows 8 have less installs than 7.

The first software needed to be dowloaded is obviusly the Python 3.5 interpreter and after its installation came the first surprise, it doesn't run because of some missing dlls; after some research I have downloaded the redistribuitable runtime and it started to work as expected.

After Python was ready I've had to install the pip utility which was not automatically installed, maybe I have forgotten to chek the corresponding option in the installer; anyway everything installed correctly using it, this is a brief list of dependencies:

  • requests: OK

  • paramiko: OK

  • imapclient: OK

  • PyQT: (surprisingly!) OK

  • cx_Freeze: Not ok at all..

cx_Freeze requires Widonws build tools that can freely (as in beer) downloaded and installed but even after that I was not able to install it from pip; fortunately some kind people provided a procompiled wheel for it but be sure to download the 5.0 version because the 4.x was not able to produce a proper executable.

Additional dependencies and some setup.py notes

Having not touched I tought that the setup script was soemwhat correct since the last build was succesful, nothing more far from the truth; the first thing has been a nice surprise, cx_Freeze recognized correctly all the dlls to use in the final package and referencing additional dlls is not needed anymore, good work cx_Freeze guys!

After the first build ihave started a cycle of try, fix retry until the application could as expected, here is a list of additional dependencis that I had to install and reference in the setup.py file:

  • appdir

  • packaging

  • atexit

appdir and atexit only need to be referenced as packages instead packaging requires some more fine tuining so I had to add this additional sub_packages to the includes settings of the build_exe_options dictionary:

  • packaging.version

  • packaging.specifiers

  • packaging.requirements

Final words

It took me a couple of hours of trial/error to be able to ship the build and I hope to not have to repeat this madness again soon; if I'll need to create a new build in the future I hope that this little post will halp me not to waste my time again.

PyQT5 and cx_Freeze Windows™ target tips

One of my preferred benefits of Python is its portability and if coupled with good libraries such as PyQT5 the possibilities are endless.

Now imagine you created a new shiny desktop application and you want to distribute it, how can you package the app for easy installation?

Say hello to cx_Freeze

cx_Freeze is a tool that can create a standalone version of your application so you can easily distribute it and the end user don't have to:

  • download your source code

  • a python interpreter

  • setup pip or easy_install, libraries and so on

Another nice feature of cx_Freeze is that it can create an installer for many different operating systems giving your application a more "professional" look.

I strongly suggest giving it a try; using one of the sample setup scripts should be enough to get started.

Some minor issues

A simple application which only depends on pure python packages usually will not give you any headaches but if you have dependencies like PyQT5 a bit of attention is required. In my case not all the required DLL were included in the installer package and that generated a strange error message that was very hard to debug but thanks to sites like StackOverflow I've found a nice fix for it. It is worth noting that linked solution is not (always?) enough but there is a quick solution (at least in my case): add the file libegl.dll to the "include_files" cx_Freeze building options.

How to test your shiny new installer

In order to test your installer and be sure that all the DLLs are incuded and your application is not "cheating" on you using system DLLs I suggest to create a clean windows installation inside a virtual machine; this way you can test your installer in a real case scenario and fix your build scripts accordingly.

Logging model changes with SQLAlchemy listeners

Logging modela changes with SQLAlchemy listeners

Scenario

Imagine you have a slew of models in your application, at some point you feel the need to log somewhere creation, modification or deletion of data belonging to these models. How to proceed without having to modify the classes one by one?

What's on sqlalchemy

SQLAlchemy (http://sqlalchemy.org) offers a couple of interesting mechanisms: the first concerns the possibility to hook to some event listeners such as before_insert, before_update, before_delete and the corresponding after_*. Additional help is provided by sqlalchemy the opportunity to work on a model after its definition by overriding the method __declare_last__. Using these facts, and assuming that you have defined a model named MyModel, if we wanted to intercept the event "after_insert" we could write the following code:

class MyModel(object):
#lets pretend to have defined our model

  def after_insert(mapper, connection, target):
    #do some stuff
    pass

  @classmethod
  def __declare_last__(cls):
    event.listen(cos, "after_insert", cls.after_insert)

Whenever an object of class MyModel will be entered into the database after_insert method will be called , passing as parameters the mapping of the model, the connection and the target is none other than the object that has just been entered into the database.

In the event that you are intercepting the creation or deletion of an object is sufficient to access its primary key to identify it in your log, but if we wanted to know which fields have been modified, with new and old values, as a result of an update it gets a little more complicated, but not too much. In fact sqlalchemy allows us, quite easily , to check the status of the fields of an object using the function sqlalchemy.orm.attributes.get_history (http://docs.sqlalchemy.org/en/latest/orm/session.html#sqlalchemy.orm.attributes.get_history). This function is called for each field, it returns an object of type History (http://docs.sqlalchemy.org/en/latest/orm/session.html#sqlalchemy.orm.attributes.History) which we will use the method has_changes() to check for changes, and if there were, getting the new and old values of the field that we are analyzing, for example:

h = get_history(target, "a_field")
if h.has_changes():
  #do something using h.deleted list to get the old values
  #do something using h.added list to get the new values

LoggableMixin

Clearly to do this for all models of an application may be costly in terms of time and code maintenance (and extremely annoying) so you might think about creating a generic Mixin with which to extend the models of our application. Below is the skeleton for the implementation of the above mixin, omitting the details of where and how the logs are stored:

class LoggableMixin(object):

  def after_insert(mapper, connection, target):
    #do some stuff for the insert
    pass

  def after_update(mapper, connection, target):
    #do some stuff for the update, maybe saving the changed fields values using get_history
    pass

  def after_delete(mapper, connection, target):
    #do some stuff
    pass

  @classmethod
  def __declare_last__(cls):
    event.listen(cos, "after_insert", cls.after_insert)
    event.listen(cos, "after_update", cls.after_update)
    event.listen(cos, "after_delete", cls.after_delete)

so, for each model we want to log changes it will be sufficient to inherit from LoggableMixin:

class MyModel(SomeSuperClass, LoggableMixin):
  pass

Improvements

One of the first improvements you can make to the class LoggableMixin could be the separation of the class in three different classes eg . LogInsertMixin, LogUpdateMixin LogDeleteMixin, in my case I preferred to have it all together given the small size of the class. A second improvement would be the generalization of mixin allowing you to specify which functions (or methods) to be assigned to different listeners, and once more the specific needs of the application I'm working on does not require this level of abstraction and can live well with this approach.

Conclusions

SQLAlchemy provides a number of services to work with the model, the system just described would not have been so easy to implement if it were not for the quality of the API of sqlalchemy. I invite anyone to go deeper in the documentation for sqlalchemy (http://docs.sqlalchemy.org) because within it are preserved gems of great value. For those wishing to see a concrete implementation of the topics discussed in this post they can take a look at the file sysgrove/models.py in the repository at https://bitbucket.org/sysgrove/sysgrove

Our QT date picker

Our QT date picker

One of the first works carried out just entering at SysGrove (http://sysgrove.com ) consisted in rewriting the date picker component in our application and requests regarding this component was that it was similar, at least as a feature, to the date picker available in windows 7 ™. The most interesting aspect of this component is the ability to be able to do some sort of zoom dates, I try to explain with an example:

  • It would be a classic date picker where are the days of the current month and the ability to change the month with the classic arrows

  • Clicking on the button in the middle of the arrows the days are replaced by months and the arrows will change the current year

  • Again by clicking on the button in the middle of the arrows months will be replaced by the years and the arrows will change the decade

  • Selecting a year will return a month view and by selecting a month will return a day view

Some images could help to understand the behaviour:

day selector

The day selector, with month changer arrows; if you click the middle button you'll see the month selector.

/images/qt-calendar/date-picker-month.png

The month selector, if you click on the middle button you'll see the year selector.

/images/qt-calendar/date-picker-year.png

The selector selector.

Fortunately SysGrove is a company that believes strongly in open source and sharing so I got permission to release this as open source project component, nice! :)

At present the appearance of the component is not that great and you have to put hand to customize the code of the component , in the future I hope to be able to allow the customization of the graphical component directly in the process of creating an instance rather than in its base code.

The source of this component for the moment can be found in the main repository of our application ( https://bitbucket.org/sysgrove/sysgrove ) path sysgrove / ui / classes / widgets / calendar.py