How I created "Melody & Memory"
Thursday, April 29th 2021
Leaving my job
I previously worked as a programmer in a software house. I really liked that job but I had a feeling deep down that I wanted to be free and work on my own things. So one day I decided it was time to pursue that dream.
I want to design and program games, specifically mobile titles. However, I know from experience that projects tend to grow in scope and time and take years to finish. I couldn't afford such commitments so I decided to build a small first project first. It would clear the path, shine light on areas that were previously unexplored and completing it would motivate me to continue on the journey. On the other hand, if I failed this small project it would be a clear signal to rethink my decision to leave my job.
Coming up with an idea
I needed something that could be completed in a matter of weeks and not months. My sister used to play Piano Tiles on her phone and I thought that playing music and pushing buttons was nice so that became the idea. The player would have to remember a sequence of button pushes and pushing the buttons would play a melody. My theory was that since the player gets both visual and audio cues it would be easier for them to remember longer patterns.
Choosing a technology
Most people in game development use engines for their games. This makes a lot of sense—an engine saves time and provides necessary tooling. I found however that I dislike game engines, because they each have their own way to do things and it always follows the hollywood principle: "Don't Call Us, We'll Call You.". Your code is never the main thing that orchestrates action, rather it is glue for functionality provided by the engine. The other thing that I dislike about game engines is that the way they do UI isn't similar to React, which for me is the most natural way of programming user interfaces.
Since the game I envisioned is 90% UI I decided to write the game in React itself. One open question was how do I make it a mobile app then? I remembered how quirky Cordova was all those years ago when I last played with it. Fortunately for me the people behind Ionic created Capacitor, which is a fork of Cordova that doesn't get in the way. So I chose that. An additional benefit of using web technologies was the ease of localization, which I believe to be a very important aspect of every software program.
First steps
The first week of development was very promising. I managed to code simple vector buttons that responded to input in a satisfying way. Contrary to what you might think it wasn't just styling html buttons with css. The default button behavior has many states and behaves unpredictably on different platforms I wanted to release on. One problem was that the click event is fired on first touch on one platform and after a small delay on the other.
Initially I wanted to use the fastclick library, but that quickly introduced even more problems and weird behavior. Finally I settled on listening on mousedown and touchdown events and calling the onClick callback manually. Additionally the button plays an animation every time it is pressed, but never stays pressed, no matter how long the user holds their finger on it. This is very important, because the length of the button press needs to match the sound played and relying on user input to keep it pressed or release it on time doesn't work.
Adding music
Of course the game wouldn't be complete without music and let me tell you, adding music was a fun ride. The first idea was to find individual mp3 files for each note on the piano and then play those mp3s on button click. Easier said than done however, as finding such files proved difficult. Next idea—find a js library for playing the piano. This is how I found MIDI.js. It uses soundfonts (like normal fonts but for sounds, and yes, my mind was blown away too) and has intuitive apis like noteOn and noteOff. Satisfied I proceeded to implement music.
But oh no. What music will I even play? I was limited not only by copyright on songs but also by the notes that I would be able to find. At first I tried to manually read music sheets and convert them into something that my little software would be able to play, but this was very time consuming and error prone. As a bonus I learned to read musical notation, but I am still very bad at it. Given that manually transcribing music was no good I did what any reasonable programmer would do. I found xml versions of music sheets for the excellent music software MuseScore and converted them into a format playable in the game. That took a bit of time but sped up level creation immensely.
Unfortunately one other problem with the music began to surface. On iOS the playback was extremely laggy. At first I thought that it was a problem with web audio on iOS in general and explored using native solutions. Dissatisfied with available options I found that there exists a generalized audio library for the web called Howler. Howler needed individual mp3s to play for each note, but now it was easy to obtain them, because they could be extracted from the soundfont. With Howler I was able to get rid of both MIDI.js and the lag.
iPhone problems continue
Knowing that I want to release on both Android and iOS I was confronted with the fact that I didn't posses an iOS device to test the game on. Relying on the simulator wasn't going to cut it, because there are a number of factors that the simulator does not accurately represent, one of them being performance.
I decided to order a used iPhone 7. One reason for a used device was of course the lower cost, but I also wanted to test on the lower end, since achieving stellar performance on a new iPhone 12 wouldn't be a challenge. Well, ordering a used device was a good idea, but being cheap about it proved a bad one. I bought the cheapest phone available and the seller wasn't even kind enough to deliver a clean device. I had to clean the sticky and dusty iPhone only to find out after one day of use that it had an iCloud lock! Returning it and getting the money back was also a lot more hassle that it should be and left me frustrated at the time wasted and inability to effectively test on iOS.
Eventually I just handed over more money to a more reputable seller and purchased a nice, used iPhone 8 that didn't have any of the aforementioned problems. Phew...
Preparing for release
Okay. The game is ready. Now what? In order to release I need to finish localizations and do screenshots. Seems easy enough, right? The fact that I can understand english, russian, polish and german definitely helps, but I wanted to additionally challenge myself to include spanish translations. I found that if approached carefully Google Translate is actually a miracle. I also found that it deals much better with longer texts, so when I translated a single word or a quick phrase I often wrapped it into a longer sentence to provide necessary context for the translation.
Once the translations were ready it came time for screenshots. 5 languages, 6 devices (4 for iOS, 2 for Android) and 3 planned panels. This amounts to 90 images, and I wanted them to not only be simple screenshots but feature device frames, distinct backgrounds and some text. As with the music doing this by hand would be a nightmare, especially considering the need to fire up the simulator for each device and also clicking through to the correct UI state.
In order to automatically generate screenshots I once again leveraged the use of web technologies. With the help of puppeteer I wrote a script that sets up correct window sizes, navigates to special routes that have hardcoded UI states and takes screenshots. Then using a publicly available device frames from Facebook i embedded each screenshot into a correct device frame. This needed a bit of manual work upfront to include status bars into the images, but it only had to be done once for each device. After all of this I used a generated svg to render the device, the screenshot, the background and some text into the final image. This might seem like a lot of work, but upon discovering any errors in the screenshots I was able to effortlessly rerun the code and generate correct versions.
Release and future plans
And that leaves us here. The app has now been approved and released on both Google Play and App Store as well as a browser release. Funnily enough it is available absolutely everywhere except of Germany, because Ravensburger has a trademark on the word "memory" in Germany and I wasn't able to release the app there.
And now for the future plans. I have been longing to work on a fully realized version of a game I created a few years ago for the js13k competition. This new version is going to have updated graphics and more interesting levels as well as general polish. Overall I have very much enjoyed this new "work" where I get to decide what I focus on. This new game is also not going to be free as I intend to start creating some revenue streams to support myself in the future.
Thank you very much for reading.
Peter