**UPDATE** I’ve just posted a new video to youtube, and embeded at the bottom of this post, showing the point cloud visualisation running in realtime.

You may know from this blog, that I work for a company called Square Enix, and before that Eidos.  SE is a video games publisher, famous for Final Fantasy, and through the acquisition of UK publisher Eidos, Tomb Raider.  Among other duties, I manage the metrics system and create data and visualisations that the business uses to make better games.  All the data we collect is anonymous. For a while now I’ve been using this data to create Heatmaps of player activity. You can see some of these on the Just Cause website.

I’ve been using Processing to create tools that render the heatmaps, but while the logical structure of program is fairly simple, there are significant challenges in working with large datasets. The primary challenge is loading the data into memory. The data is all held in a SQL database, and while I could connect to the DB directly using processing, the DB is optimised for data-in operations, not data-out, so you don’t want to be pulling the data out too often. Instead, I dump the raw spatial data (X,Y,Z coordinates) into a CSV file, one record per row. I usually create heatmaps from datasets in excess of 1 million rows, and most of them are between 5 – 20 million rows (I have one that is 22 million rows!). A CSV file containing 10 million rows of spatial data is about 364MiB in size (the 22.3m row CSV is 802MiB!). In order to create the data structures in memory to hold sets this large, I have to work in 64bit mode to get over the Windows 32bit memory restrictions.

When I wrote the tool for heatmapping, I was pretty fresh to processing, and I can now see many ways to optimise this process to be more efficient at memory usage, but I’m not great at refactoring, and as it works, I’ve not been back to optimise it. The heatmapper application works by looking at the resolution of the image you want to create (eg. 2000x2000px) and then subdividing this into a cell structure whose size determined by the “cell resolution” variable. The cells are essentially buckets, into which I put the rows of raw data. Each row of raw data (the data’s X and Y value) is processed to find out which bucket that row of data fall in and then a Cell’s value is incremented by one every time an element of the raw data lands in it.

A heatmap of more than 22.3 million player 'Extraction' events from Just Cause 2

Once the whole raw dataset has been parsed, and the Cells populated with counts, we render the cells. This is a simple process of drawing a circle of the same diameter as the cell size. The colour of which is determined by the number of the raw data elements that landed in that cell. The palette is scaled so that the cell with the largest value of counts is always pure white, so there is no clipping in the colour (although you can remove this scaling, meaning that any cell with more than 255 counts in it will be white, the trade off is increased colour resolution for the less populated cells). I use a simple alpha blend to help smooth out the rendering. The results are pretty good.

What was bugging me was that I was getting this huge set of raw data, but only using two thirds of it. I never ventured into third dimension by factoring the Z coordinate (the ‘height’ spatial component) into the process. At the beginning of this year, I decided to do something about this. I started a project to rebuild the heatmapping application in 3D.

My first results we truly awful. I rendered 9000 points in rows using the OpenGL renderer, with colour to show the relative height of the points. With only 9k rows, I couldn’t manage more than a couple of frames a second. The culprit was the Map() function that I was using for scaling the colour palette, for each point, each frame. Once I removed that, Life got a lot better, but I couldn’t render more than a few thousand points without the framerate falling off a cliff.

Early rendering of a point cloud from the JC2 Data

It became obvious that I would need to start getting a bit more low level with OpenGL to get more performance. This is where I discovered the work of Andrés Colubri and his GLGraphics library for Processing. With some great support through the Processing Forums, I was able to move a lot of the heavy lifting from the CPU to the GFX card, and ultimately create a vertex buffer object holding the points. I can render 11 million points in realtime at 30fps, on my desktop computer. This was several orders of magnitude greater performance than I ever thought would be possible.

The JustCause2 dataset that I was using was a selection of player deaths specifically where the player had died from an impact event. This was great because players tend to spend a lot of time jumping off tall buildings or riding around in helicopters and planes, so impact in this context is generally impact with the geometry of the environment. When the data is rendered, you can see the underlying world, almost as clearly as if we were rendering the 3d mesh itself.

Finally, and arguably the most important aspect of any data visualisation is working out the best way to communicate it. The data looked great, but I couldn’t distribute it as source because of the size of the CSV files that accompany it. I needed to some way of taking the viewer through the data and showing them points of interest. Up stepped another Processing hero, Jean Pierre Charalambos and the ProScene library. With ProScene, and Jean Pierre’s help I was able to direct the camera around to create a visual tour of the data which I then rendered out as individual 1080p frames, and assembled later into an animation.

It’s not over, in fact this journey into 3D visualisation of big data is only just starting. The JC2 Point Cloud animation isn’t really a heatmap at all, more like a 3D scatter graph of points. I will be investigating how to build a 3D cellular structure next, to create a true study of interactions in the 3D space. I will most likely be using data from the upcoming Square Enix game, Deus Ex: Human Revolution as it’s level based structure is more suitable for this project than the vast open world of Just Cause 2.

Click here to view the embedded video.

He was asked during the Q&A if we would ever see Elite again, to which he just replied ‘Yes’. Of course that’s nothing new, we’ve known for years that Elite is supposed to be in active development, but he simply won’t be drawn on any details. My first thought was that i hope he uses Eve as a graphic reference, but beats Eve into a bloody mess by putting some real ship control and combat in there, rather than eve’s lame point and click navigation and warfare.  My second thought which was much sadder was that whatever he does, it’s never going to meet our expectations.

One thing that did freak me out a little was when he mentioned something that addressed a deep sense of inadequacy I’ve carried since being a child.  Even friends who only dabbled with Elite were able to stumble across one or more of the mythical missions, however I never did.  Despite achieving a rank of dangerous, I never had so much as a suggestion of a mission.  How could I ever pretend to be worthy of my commander rank?  Of course, it was obvious. I was playing on an Acorn Electron, and the missions were the bit that was chopped to shoehorn the game into the reduced memory space.  I can’t tell you how much of a relief it was to hear that!

Click here to view the embedded video.

I’d also forgotten how painful it must of been to play, it took 20 mins to load it in by tape! This, according to Braben, allowed for a degree of forgiveness by the players for the very steep learning curve.  It may have been hard to play if you were new to it, but after waiting 20mins for it to load, you’d at least give it a few mins of playing to see if you could master it.  I’m sure that’s a trick being used on some PS3 games today

I can’t believe I’m going to say it but… I think the iPhone is going to be a great gaming device. Damm Apple and all their ultra popular design. I was turned off touch screen by a generation of weak underpowered devices who didn’t have enough processor grunt to refresh the touch and update the screen, let alone run any games, but then I saw the LG Viewty. At last there was a touch screen device that responded to a touch with out having to wait a week for anything to happen. Nearly. Then I relented and allowed myself to be in the same room as an apple device for a short time, and found that I was deeply impressed. The screen is huge, both in resolution and real estate and the touch is so responsive, it almost feels predictive. The motion sensor works and the multi touch pinch is fun. This could be a great gaming platform. I’ve never been at a show where there was more anticipation for one technology release, namely the official iPhone SDK (now delayed untill march).

The main event also had it’s moments. I have been thinking a lot about PS3 Home, and I can’t help thinking that it’s not going to be a mediocre so-so. It will either be the most laughable attempt at community ever or a brilliant success. My game of the show was Chat Noir which I thought was perfect in every way. The experimental gameplay sessions, was terribly disappointing except for a few notable highlights, namely Ricky showing off Bernie, which was brilliant and a preview of the awesome Crayon Physics Deluxe. Where was the indie game jam? The whole event seemed to be lacking the energy of previous years, it was the first time in 6 years that I’ve seen empty seats in that room.

It rained a lot too.

Click here to view the embedded video.