Screentendo

Screentendo is a desktop application that allows you to turn a selection of your screen into a playable level of Super Mario Bros.

I’ve never built a Cocoa app or used Sprite Kit before, so it was a great excuse to play with both technologies. You can get the (very hacky) source code here.

How does it work?

When Screentendo is launched, a semi-transparent window appears which can be moved and resized over other application windows. After placing the Screentendo window over an area of the screen, clicking inside the window will cause the app to render a game level based on the content beneath it.

There are two basic steps to Screentendo; image processing to determine the structure of the target region, and the level generation.

Image processing

The first step is to fetch the target region underneath the Screentendo window. The following shows an example targeting a chart within Google Sheets (in Safari):

  • The app uses the CGWindowListCopyWindowInfo API (from the Quartz Window Services) to get a list of windows in the users current session, in the order in which they appear on the screen
  • From this list of windows, the application fetches the window data (window ID, bounds, size, etc) for the Screentendo window and the second window in the hierarchy (the Safari window covered by Screentendo)
  • With the target window information, CGWindowListCreateImage is used to retrieve a screenshot of Safari
  • The screenshot is then cropped to the bounds of the Screentendo window, using the difference between the Safari window origin and the Screentendo window origin (as shown below), and the Screentendo window’s width and height:

screentendo_target_window

Screentendo overlaying target application (chart in Google Sheets)

cropped

Cropped image


The cropped image is then passed through a number of image filters, before finally being converted into a format that Screentendo can use to render a game level:

Motion blur – this helps to reduce the impact of general noise and small visual artefacts in the image:

motion_blurred

Motion-blurred image


Luminance filter – the average luminance threshold of the image is calculated, and then reduced to the two colours based on the calculated threshold:

luminance_filtered

Luminance-filtered image


Pixellation filter – a pixellation filter is applied to simplify the image detail, and prepare it for sub-blocking:

black_and_white_image

Black and white pixellated image


Sub-blocking
– the image is then split into sub-blocks (by default, 10×10 pixels):

subblocked

Sub-blocked image

 

subblocked_zoomed

Sub-blocked image (enlarged)


Average block colour to array
– for each sub-block in the image, the average colour of the block is calculated. A 2-dimensional array is created, and any sub-block with an average colour which is mostly black is set to ‘1’, and any sub-block with an average colour which is mostly white is set to ‘0’:

subblocked_array

2-dimensional array representing the image block structure (enlarged)

Level generation and game logic

The 2-dimensional array representation of the image is passed to the GameScene class, which is responsible for generating the game level.

Iterating over the array, any values of ‘1’ are generated as blocks, and ‘0’ values are ignored:

mid_generating

Blocks during level generation

completed_generating

Blocks, cloud, background and player sprites

When the array has been processed, the background, clouds, and player sprite are then added to the scene. The rest of the game play relies on a basic physics implementation based on the Sprite Kit physics engine (to handle player physics, detect collisions, animate flying block debris, etc).

The app also has a menu option to change the block size (small, medium, large). A smaller block size increases the resolution, but takes longer to process.

Limitations

This app is a proof-of-concept hack, and has a few shortcomings. Image processing is currently really (really) slow – sub-blocking the image takes a long time (each sub-block is an NSImage, which is a pretty inefficient way of solving this problem, but quick to implement). The current implementation also requires a reasonably distinct contrast in the underlying image for the block detection to work. Finally, the physics is a little screwy – I didn’t set out to write a Super Mario Bros emulator, just something that would work “well enough” – and as such, there are some issues with ghost vertices (particularly with vertical walls) that I didn’t get round to resolving.

Downloads

The code is available over at the GitHub repo.

Like this? Follow me on Twitter!

71 Comments

  1. Aw, very nice idea! Love it!

  2. Lucas

    Completely useless, but really sweet! Love it!

  3. This is seriously awesome. Can you make the compiled version available? I’d love to play with this, but I don’t have XCode.

  4. drfarr

    so good!

  5. Nikhil Baliga

    Brilliant 😀 I loved the idea

  6. Jagtesh Chadha

    This is awesome! Please post a compiled binary online 🙂

  7. Someone

    Please create a Homebrew repository for this application! http://brew.sh

  8. This is amazing, please make it available for download. I love mario alot and this like an amazing news.

  9. Ninja

    How will you add a goal, powerups, and enemies? What would have to be in the image to be qualified as a powerup or enemy?

Trackbacks for this post

  1. Este app transforma qualquer parte da sua tela em uma fase jogável de Super Mario | Tech News
  2. Este app transforma qualquer parte da sua tela em uma fase jogável de Super Mario |
  3. Este app transforma qualquer parte da sua tela em uma fase jogável de Super Mario | Rhuan Gonzaga
  4. Screentendo | Das Kraftfuttermischwerk
  5. Screentendo Turns Your Desktop Into a Playable Mario Level - eTeknix
  6. Screentendo can turn almost anything on your computer into a Mario level | 381test
  7. Screentendo can turn almost anything on your computer into a Mario level | TechDiem.com
  8. Screentendo can turn almost anything on your computer into a Mario level | I World New
  9. Screentendo can turn almost anything on your computer into a Mario level | SomeLearning
  10. Screentendo can turn almost anything on your computer into a Mario level | BawlBuster
  11. Screentendo can turn almost anything on your computer into a Mario level | Game Developr
  12. Screentendo can turn almost anything on your computer into a Mario level | Ad Pub
  13. Screentendo can turn almost anything on your computer into a Mario level – VentureBeat | IT Manager Resume
  14. Screentendo can turn almost anything on your computer into a Mario level – VentureBeat | IT Director Resume
  15. Screentento verwandelt Screenshots in spielbare Super Mario Level • ZWENTNER.com
  16. Noticias – Screentendo, la aplicación que convierte imágenes en niveles de Super Mario Bros
  17. Screentendo, la aplicación que convierte imágenes en niveles de Super Mario Bros | Blog de Alcocer Sotil
  18. Game-News: Screentendo verwandelt euer Hintergrundbild in ein Level von Super Mario - SHOCK2
  19. Webmediaplus – Convierte cualquier captura de pantalla en un nivel de Super Mario con Screentendo
  20. Screentendo can turn almost anything on your computer into a Mario level | http://buymygamescheap.com/
  21. Screentendo Lets Players Create Mario Levels From Images | TechNewsDB
  22. screentendo, transforma cualquier parte de tu pantalla en un nivel de Super Mario Bros
  23. screentendo, transforms any part of your screen to a level of Super Mario Bros | Cnn Hit New
  24. screentendo, transforma cualquier parte de tu pantalla en un nivel de Super Mario Bros |
  25. Handy Vandal's Almanac » Blog Archive » Turn your screen into a playable Super Mario level - Resources for Game Designers
  26. Nerdcore › Screentendo: Super Mario-Levels generated from Desktops
  27. Screentendo: aplicativo transforma tela do Mac OS em fase de Mario [vídeo] - Baixamais Downloads - Artigos e notícias
  28. Screentendo: aplicativo transforma tela do Mac OS em fase de Mario [vídeo] | Irmandade Ômega
  29. Desktop App Turns Any Part Of The Screen Into Super Mario Bros. | The Root Shell
  30. Screentendo can turn almost anything on your computer into a Mario level | Trend Inside
  31. Turn anything on your desktop into a playable level of Super Mario Bros. | exciting games on mac
  32. Turn anything on your desktop into a playable level of Super Mario Bros. | Best Gamers News
  33. Turn anything on your desktop into a playable level of Super Mario Bros. | Android and IOS News
  34. Turn anything on your desktop into a playable level of Super Mario Bros.
  35. Screentendo transforme capture d’écran en jeu Super Mario Bros
  36. Turn Your Computer Screen Into A Playable ‘Super Mario Bros.’ Level - Kreative Feed
  37. Turn Your Computer Screen Into A Playable ‘Super Mario Bros.’ Level | A multi-disciplined Graphic Designer and Art Director based in Bristol with a passion for technology and digital design
  38. Screentendo, trasforma il tuo desktop in un livello di Super Mario Bros. #LegaNerd
  39. Turn anything on your desktop into a playable level of Super Mario Bros. - seekape
  40. Turn anything on your desktop into a playable level of Super Mario Bros. - Virtual Reality
  41. Turn anything on your screen into a playable Super Mario Bros game | Bobbie Hollowell Blog
  42. Turn anything on your screen into a playable Super Mario Bros game | 3Buzz Now
  43. App turns your desktop into Super Mario level - Pixel Dynamo
  44. Turn anything on your desktop into a playable level of Super Mario Bros. | GAMES
  45. Transforme qualquer coisa do seu Desktop em um level jogável de Super Mario Bros. - Geek Vox
  46. Screentendo, trasforma il tuo desktop in un livello di Super Mario Bros. - Studio Citeroni | E-commerce, Siti Web e SEO
  47. Convierte cualquier cosa de tu computadora en un nivel de ‘Super Mario Bros’
  48. iCulture Vandaag: dit was woensdag 27 mei 2015
  49. Desktop App Turns Any Part Of The Screen Into Super Mario Bros. | Kotaku Australia
  50. Make a Mario Level Out of Your Screen With One Click | AQOVD's BLOG(بلوق)
  51. Transform parts of your screen into Mario levels with Screentendo | The Daily Ding
  52. Make a Mario Level Out of Your Screen With One Click - Game News
  53. با Screentendo برای سوپر ماریو مرحله بسازید! - بازی تاک
  54. Heard on Air: News Links and Videos « CBS DC
  55. Programmer finds work for idle screens | Liblog
  56. Screentendo | the Brick In the Sky
  57. Play your Desktop! |
  58. HOW TO PLAY SUPER-MARIO IN ANY PART OF YOUR SCREEN | Mac User - MIX NEWSMIX NEWS
  59. Screentendo Turns Desktop Screens Into Games Of Super Mario Brothers – So Cool! | Oh Doozy
  60. Awesome MacOs – Massive Collection Of Resources – You Ever Wondered To Get! – Learn Practice & Share