Screentendo

Screentendo is a desktop application that allows you to turn a selection of your screen into a playable level of Super Mario Bros.

I’ve never built a Cocoa app or used Sprite Kit before, so it was a great excuse to play with both technologies. You can get the (very hacky) source code here.

How does it work?

When Screentendo is launched, a semi-transparent window appears which can be moved and resized over other application windows. After placing the Screentendo window over an area of the screen, clicking inside the window will cause the app to render a game level based on the content beneath it.

There are two basic steps to Screentendo; image processing to determine the structure of the target region, and the level generation.

Image processing

The first step is to fetch the target region underneath the Screentendo window. The following shows an example targeting a chart within Google Sheets (in Safari):

The app uses the CGWindowListCopyWindowInfo API (from the Quartz Window Services) to get a list of windows in the users current session, in the order in which they appear on the screen
From this list of windows, the application fetches the window data (window ID, bounds, size, etc) for the Screentendo window and the second window in the hierarchy (the Safari window covered by Screentendo)
With the target window information, CGWindowListCreateImage is used to retrieve a screenshot of Safari
The screenshot is then cropped to the bounds of the Screentendo window, using the difference between the Safari window origin and the Screentendo window origin (as shown below), and the Screentendo window’s width and height:

Screentendo overlaying target application (chart in Google Sheets)

Cropped image

The cropped image is then passed through a number of image filters, before finally being converted into a format that Screentendo can use to render a game level:

Motion blur – this helps to reduce the impact of general noise and small visual artefacts in the image:

Motion-blurred image

Luminance filter – the average luminance threshold of the image is calculated, and then reduced to the two colours based on the calculated threshold:

Luminance-filtered image

Pixellation filter – a pixellation filter is applied to simplify the image detail, and prepare it for sub-blocking:

Black and white pixellated image

Sub-blocking – the image is then split into sub-blocks (by default, 10×10 pixels):

Sub-blocked image

Sub-blocked image (enlarged)

Average block colour to array – for each sub-block in the image, the average colour of the block is calculated. A 2-dimensional array is created, and any sub-block with an average colour which is mostly black is set to ‘1’, and any sub-block with an average colour which is mostly white is set to ‘0’:

2-dimensional array representing the image block structure (enlarged)

Level generation and game logic

The 2-dimensional array representation of the image is passed to the GameScene class, which is responsible for generating the game level.

Iterating over the array, any values of ‘1’ are generated as blocks, and ‘0’ values are ignored:

Blocks during level generation

Blocks, cloud, background and player sprites

When the array has been processed, the background, clouds, and player sprite are then added to the scene. The rest of the game play relies on a basic physics implementation based on the Sprite Kit physics engine (to handle player physics, detect collisions, animate flying block debris, etc).

The app also has a menu option to change the block size (small, medium, large). A smaller block size increases the resolution, but takes longer to process.

Limitations

This app is a proof-of-concept hack, and has a few shortcomings. Image processing is currently really (really) slow – sub-blocking the image takes a long time (each sub-block is an NSImage, which is a pretty inefficient way of solving this problem, but quick to implement). The current implementation also requires a reasonably distinct contrast in the underlying image for the block detection to work. Finally, the physics is a little screwy – I didn’t set out to write a Super Mario Bros emulator, just something that would work “well enough” – and as such, there are some issues with ghost vertices (particularly with vertical walls) that I didn’t get round to resolving.