**UPDATE** I’ve just posted a new video to youtube, and embeded at the bottom of this post, showing the point cloud visualisation running in realtime.

You may know from this blog, that I work for a company called Square Enix, and before that Eidos.  SE is a video games publisher, famous for Final Fantasy, and through the acquisition of UK publisher Eidos, Tomb Raider.  Among other duties, I manage the metrics system and create data and visualisations that the business uses to make better games.  All the data we collect is anonymous. For a while now I’ve been using this data to create Heatmaps of player activity. You can see some of these on the Just Cause website.

I’ve been using Processing to create tools that render the heatmaps, but while the logical structure of program is fairly simple, there are significant challenges in working with large datasets. The primary challenge is loading the data into memory. The data is all held in a SQL database, and while I could connect to the DB directly using processing, the DB is optimised for data-in operations, not data-out, so you don’t want to be pulling the data out too often. Instead, I dump the raw spatial data (X,Y,Z coordinates) into a CSV file, one record per row. I usually create heatmaps from datasets in excess of 1 million rows, and most of them are between 5 – 20 million rows (I have one that is 22 million rows!). A CSV file containing 10 million rows of spatial data is about 364MiB in size (the 22.3m row CSV is 802MiB!). In order to create the data structures in memory to hold sets this large, I have to work in 64bit mode to get over the Windows 32bit memory restrictions.

When I wrote the tool for heatmapping, I was pretty fresh to processing, and I can now see many ways to optimise this process to be more efficient at memory usage, but I’m not great at refactoring, and as it works, I’ve not been back to optimise it. The heatmapper application works by looking at the resolution of the image you want to create (eg. 2000x2000px) and then subdividing this into a cell structure whose size determined by the “cell resolution” variable. The cells are essentially buckets, into which I put the rows of raw data. Each row of raw data (the data’s X and Y value) is processed to find out which bucket that row of data fall in and then a Cell’s value is incremented by one every time an element of the raw data lands in it.

A heatmap of more than 22.3 million player 'Extraction' events from Just Cause 2

Once the whole raw dataset has been parsed, and the Cells populated with counts, we render the cells. This is a simple process of drawing a circle of the same diameter as the cell size. The colour of which is determined by the number of the raw data elements that landed in that cell. The palette is scaled so that the cell with the largest value of counts is always pure white, so there is no clipping in the colour (although you can remove this scaling, meaning that any cell with more than 255 counts in it will be white, the trade off is increased colour resolution for the less populated cells). I use a simple alpha blend to help smooth out the rendering. The results are pretty good.

What was bugging me was that I was getting this huge set of raw data, but only using two thirds of it. I never ventured into third dimension by factoring the Z coordinate (the ‘height’ spatial component) into the process. At the beginning of this year, I decided to do something about this. I started a project to rebuild the heatmapping application in 3D.

My first results we truly awful. I rendered 9000 points in rows using the OpenGL renderer, with colour to show the relative height of the points. With only 9k rows, I couldn’t manage more than a couple of frames a second. The culprit was the Map() function that I was using for scaling the colour palette, for each point, each frame. Once I removed that, Life got a lot better, but I couldn’t render more than a few thousand points without the framerate falling off a cliff.

Early rendering of a point cloud from the JC2 Data

It became obvious that I would need to start getting a bit more low level with OpenGL to get more performance. This is where I discovered the work of Andrés Colubri and his GLGraphics library for Processing. With some great support through the Processing Forums, I was able to move a lot of the heavy lifting from the CPU to the GFX card, and ultimately create a vertex buffer object holding the points. I can render 11 million points in realtime at 30fps, on my desktop computer. This was several orders of magnitude greater performance than I ever thought would be possible.

The JustCause2 dataset that I was using was a selection of player deaths specifically where the player had died from an impact event. This was great because players tend to spend a lot of time jumping off tall buildings or riding around in helicopters and planes, so impact in this context is generally impact with the geometry of the environment. When the data is rendered, you can see the underlying world, almost as clearly as if we were rendering the 3d mesh itself.

Finally, and arguably the most important aspect of any data visualisation is working out the best way to communicate it. The data looked great, but I couldn’t distribute it as source because of the size of the CSV files that accompany it. I needed to some way of taking the viewer through the data and showing them points of interest. Up stepped another Processing hero, Jean Pierre Charalambos and the ProScene library. With ProScene, and Jean Pierre’s help I was able to direct the camera around to create a visual tour of the data which I then rendered out as individual 1080p frames, and assembled later into an animation.

It’s not over, in fact this journey into 3D visualisation of big data is only just starting. The JC2 Point Cloud animation isn’t really a heatmap at all, more like a 3D scatter graph of points. I will be investigating how to build a 3D cellular structure next, to create a true study of interactions in the 3D space. I will most likely be using data from the upcoming Square Enix game, Deus Ex: Human Revolution as it’s level based structure is more suitable for this project than the vast open world of Just Cause 2.

Click here to view the embedded video.

Click here to view the embedded video.

Click here to view the embedded video.

Click here to view the embedded video.

Click here to view the embedded video.

If you know of any more MITO videos, please leave them in the comments.

Clusters of Stuxnet infections. Courtesy of Symantec. (Via Wired ThreatLevel)

Of course the first place to start would be the wikipedia article on Stuxnet, which does a great job of covering all areas of the discussion. It has an awesome list of references. It would probably be worth checking out the wikipedia articles on the following subjects too, as they will crop up frequently in the following discussions.

• zero-day attacks
• rootkit
• Programmable logic controller (PLC)

The best commentary I’ve found on the subject comes (bizarrely) from the US magazine, Vanity Fair. It’s not so odd when you consider that it’s published by Condé Nast who also publish Wired, and probably share staff writers and content. The article is called “A Declaration of Cyber-War” and if you read nothing else, then this will give you the big picture (although it’s a little bit sensationalist at times).

That is not to say that Wired doesn’t have it’s own coverage of Stuxnet. As you would expect it’s covered in great detail in many of the magazines online channels. The Wired DangerRoom Blog (on US national Security) has an excellent article here on Stuxnet (With Stuxnet, Did The U.S. And Israel Create a New Cyberwar Era?) and the ThreatLevel Blog (Online Security) has ongoing coverage under the Stuxnet Tag (including this particular post (Surveillance Footage and Code Clues Indicate Stuxnet Hit Iran) that discusses the origins and target(s)).

There are many interesting resources on youtube, mainly from the big anti-virus corporations discussing the exploits used by Stuxnet and it’s code structure. the only one I’ll embed here is the talk by Microsoft Engineer, Bruce Dang who (apparently) led the research into the worm’s attack on windows. It’s taken from 27C3 (27th Chaos Communication Congress) and weighs in at nearly an hour long, but is worth every minute.

Click here to view the embedded video.

Some of the other youtube videos worth looking at are:

• Stuxnet: How It Infects PLCs by Symantec
• Stuxnet/Windows shortcut zero-day explained by SophosLabs
• Interview with Ryan Singel from Wired on Attack of the Show
• Al Jazeera report on Stuxnet

There are plenty more videos on youtube on the subject of Stuxnet.

The BBC also has some good coverage, and recently reported (4th March 2011) on claims that the US and Israel were behind Stuxnet. They also earlier (15th Feb 2011) provided commentary on the Symantic report on the spread of the worm and it’s different waves of infection. (Check out the list of related stores at the bottom of the BBC articles for more of their coverage).

In all, there is enough content out there to keep me in reading and research for years to come. I think it’s only really when you get the story from lots of different angles that the you start to see the common truths emerging. To paraphrase the closing statement of the Vanity Fair article on Stuxnet, I think it really will be viewed as a Hiroshma-scale event, marking the start of a global cyber-war arms race that moves beyond simply stealing millions of hotmail passwords, to an era of targeted deployment of sophisticated digital weapons against physical infrastructure.

Olympic Pipeline Screenshot from Wired's Threatlevel Blog

This is Astronaut Tracy Caldwell Dyson looking out of the observation window, known as the Cupola on the ISS, shortly before her return to earth after a 176 day mission.

This is Jane Austin’s Sense and Sensibility. Like most of the other tree maps I’ve ordered this using a Pivot by Split Size

Pivot-by-split-size selects the pivot that will split the list into approximately equal total areas. With the sublists containing a similar area, we expect to get a balanced layout, even when the items in one part of the list are a substantially different size than items in the other part of the list. (source)

The same data can be ordered in a more conventional way; this is Sense and Sensibility  in a “Squarified Layout”:

Here are some other books, for your viewing pleasure.

Pride and Prejudice

Harry Potter and the Sorcerers Stone

2001 A Space Odessey

The Hitch-Hikers Guide to the Galaxy

Neuromancer (again!)

…in ‘Case’ you were wondering, it’s Neuromancer, by William Gibson

But…. that’s not the interesting bit.  It looks like Jonathan Koren (Robotmonkeys.net resident blogger) and I have a shared interest in Number Stations.  In fact that’s perhaps unfair on Jonathan, as he’s got far more info on them than I’ve seen in while, I’ll have to catch up on what’s happening in the murky world of global short-wave espionage.  I first read about number stations back in 2000 and thought they were the coolest thing ever.  In fact it was the seed that started me on the route to classified satellite tracking (something else I like to think I’m ‘active’ in doing, but actually just talk about a lot).  Kudos to Robotmonkeys.net for helping to keep number stations alive in the blogosphere! Yay!

Perhaps this is the source of the buzzer? (kspzel.livejournal.com via Robotmonkeys.net)

Why the curator felt that my modest creation should be included in this digital museum is something I can only speculate on, but I’m very happy that it did.  Perhaps it will inspire me to finish the project!

The Messenger (via Spime.org)

Click on the image to get the awesome full resolution version of the whole of the UK.

[from: http://www.ouramazingplanet.com/infographic-tallest-mountain-to-deepest-ocean-trench-0249/]

Another Infographic on the same subject details the current cementing plan that is favoured by BP.

[from: http://magamaps.com/?p=1477]