Hybrid Storage Mode - DataNow 4.1+

Version 2

    Verified Product Versions

    File Director 4.1File Director 4.2File Director 4.3


    DataNow 4.1 introduces a change to the way in which we store operational data, specifically user session tokens and notifications







    Session tokens:


    In DataNow, when users contact the appliance for the first time from a DataNow client, they are challenged for credentials. This is either handled silently in the form of SSO, or via a username and password physically input where SSO is not configured.


    Once the client has been successfully authenticated by an appliance, DataNow issues the client a session token that the client can tag on to any request (uploads / downloads / listing requests etc) to avoid having to re-authenticate every time.


    This session token is valid for 24 hours






    Notifications are a propriatary DataNow mechanism designed to improve the speed of propagation of changes made to content in a shared map points by different users to their local cache.


    For example, user A adds a new file to a map point called 'shared'. This generates a 'notification' which is stored by the DataNow appliance. Connected DataNow clients poll the appliance at regular intervals (30 seconds) to request any changes. If any have been registered (in this case, a new file has been created) the client will then trigger a download of this file to ensure its cache is up-to-date.


    In the absense of notifications, new content on the map point is discovered by clients


    i. When they log on and perform a login sync


    ii When the user interacts with the location in their local cache containing the new content (eg opens explorer, refreshes the screen or performs a sibling action like creating a new file)




    Prior to 4.1


    In DataNow, prior to 4.1 session tokens were held in memory by the appliance that issued the session token. In clustered environments behind a load balancer, this meant that if an appliance was taken offline, all users that appliance was servicing would be redirected to alternate appliances where they would have to re-authenticate to get new session tokens, which generates large load on the platform.


    Notifications were stored in the database (internal postgres or external SQL depending on the deployment) - as by default a notification is created for every action by every user (write notifications) and read by every connected client every 30 seconds, this mechanism was the result of the highest factor of operational load on the SQL database




    4.1 and above


    In DataNow 4.1 and above, both session tokens and notifications are stored in the Fission2 clustering mechanism.


    The net effect of this means that:


    - Operational usage of external SQL database is drastically reduced (although our sizing recommendations have not changed until which point we have adequate performance data from ongoing testing)


    - Session tokens are replicated between appliances. This means that if an appliance in the cluster is taken offline, users that had been authenticated by that appliance will have their session tokens honoured by other appliances that they are redirected to.


    The exception to this is following a patching event, whereby all appliances are disconnected/rebooted and session tokens across the estate are cleared. For this reason we recommend patching be performed in a low-utilisation maintenance window, and all appliances be taken administratively offline, and bought online synchronously to avoid uneven loading on the platform.