A journey in birdsong
“The island is as intriguing as I imagined, there are so many birds here to find and catalog! Their calls are stranger than anything I've ever heard. There is much to explore!
It will be morning soon, and I cannot imagine the glory of the dawn chorus!”
Watch Menura's dev diary of the experience HERE.
In Menura (from Menura novaehollandiae or “Superb Lyrebird”), you play an ornithologist sent to a mysterious island to catalog the bird species that live there. Based on the childhood fascination with birdsong of one of our team members, we imagined a forest in which players could casually wander, surrounded by birdsong and the sounds of nature.
We are Nekologic, a game studio based in Kyoto, Japan, and it’s our pleasure to introduce the secret inner workings of our game, developed with Dolby Atmos and Wwise, for the Step Up Your Sound Jam 2023 with Playcrafting.
Evoking a feeling
We started with a feeling that we wanted to evoke: a blend of relaxation, exploration and discovery. We felt we could use Dolby Atmos to create an immersive soundscape that could be experienced even without visuals. The bird theme naturally suggested a forest setting, but to retain a feeling of wonder–and to maintain an emphasis on the sonic aspect of the game–we decided to use an abstract visual language, describing the scene with color and movement. To emphasize the feeling of the natural world, we wanted everything to be organically generated, so the player could constantly discover new bird species, and the bird songs for each species would be recognizable yet never repetitive. The final piece of the puzzle was the passing of time, which would allow us to vary which birds appeared–and how they behaved–at different times of day.
Gameplay design
We decided early on that a first-person perspective would allow the player the best chance to experience the feeling of moving through and looking around the forest. Each bird would be placed in 3D space–hiding in trees and bushes–and the spatial audio provided by Dolby Atmos would allow the player to pinpoint where the birds were hiding. Some of the trees are relatively tall, and since Dolby Atmos is particularly good at rendering the difference between sounds overhead and on the ground, this is a perfect scenario for a full 3D experience with a good depth of the mix. When the player is close enough, he or she would be able to tap on the screen to take a picture of the bird to place in their ornithologist’s notebook, along with a recording of the bird’s unique call.
There were a few technical challenges in executing this idea:
- How to use Dolby Atmos and Wwise to generate the birdsong
- Where the birds would be and when they would sing
- A system for the bird’s appearance in the logbook
- A relaxing and intriguing environment, varied in terrain and vegetation
- Controlling the randomness of all of the above
Generative Birdsong with Dolby Atmos and Wwise
Our idea was to create an infinite variety of distinct bird voices that not only imitated the sound of real-world birds but also went beyond, towards the supernatural. Each bird species has its own vocabulary of song elements consisting of notes, swoops, chirps, trills, warbles and other features. Individuals of that species then recombine these elements spontaneously when singing based on environmental factors.
The song elements are modeled as sequences of instructions passed into Wwise as gradually changing Real Time Parameter Controls (RTPCs) with different interpolation shapes and durations. Since real-world bird songs contain an incredible amount of detail in fast-paced modulations and timbral variations, we knew that we needed to model the majority of these modulations directly within Wwise to avoid overloading the Unity update loop. After some experiments with Wwise’s built-in synthesizer plugins–using simple sine oscillators and nested modulator signals of different shapes (see figure below)--we decided to use actual audio fragments of songs of different birds, each only a few milliseconds long, to provide a more realistic variety of timbres. We play these samples using looped Wwise sound sources equipped with LFO modulators, whose parameters (e.g. shapes, frequencies, depths, etc) are in turn controlled by RTPCs.
We then randomly generated RTPC ramps for each species based on constraints such as characteristic ranges for pitch, modulation frequency and depth, element durations, singing variety, and recombinations. Due to Dolby Atmos’s rendering clarity–especially of overhead sounds–this produced a satisfying soundscape where a large number of different bird species were distinguishable. In the current build, the settings of the constraints are still kept relatively tame. But for the next iteration, our model will generate much longer, wilder, and more complex songs.
A book of birds
To show the birds in the book, we designed a system affectionately known as “The Frankenbirber”. Each part of the bird was created as a png using the “value” from the hue/saturation/value color space to allow us to apply different colors later. These parts were then built into Unity prefabs. When creating a new bird species, the system picks a prefab for one of each part and combines them using anchor points held in the prefab. It then introduces variations in size and rotation to give each bird a unique personality. To add color, we used the 10/30/60 rule to create harmonious palettes with a nice balance between the accent, secondary, and main colors. We then picked 2 colors for each part of the bird using rules to decide where colors would be placed. We favored accent colors for plumage and tails, using secondary colors more on the body and wings and so on. Each color was further modulated to give a slight but noticeable variation from the base color.
The Island
Early on, we debated whether to allow the player to roam through a boundless forest. Given the time constraints, we decided that this would be too big of a challenge. Instead, we chose to place the player on an island, which created a natural barrier and also supplied relaxing sounds of the sea. It also fit well with the emerging artstyle of bright, clear colors and a narrative that placed the player as a “Darwin in the Galapagos Islands” style ornithologist.
Randomly generating the island allowed for endless chances at a new place to explore and get lost in. We first created a system to deform and tile meshes to create the natural land formations, allowing for future optimization for larger islands as required by rendering only the tiles directly around and in view of the player at any given time. We then defined sets of biomes–which consist of both terrain coloration and flora–along with distribution weights to determine the density of each plant and tree. Lastly we deform the island’s mesh a second time to give it a nice dome-like slope from the center of the island down to the sea, as well as add interest to the coastline. In the future, we’d love to add more detail to this, as well as weather patterns and interplay between them and biomes, and then open these tools up to the player for customization of their own island experience. It will be exciting to see what players come up with!
Randomness
Finally, we needed something to handle both generating nice ranges of random numbers and also being able to replicate a particular island or bird species for multiple players and playthroughs. We created a wrapper for C#’s System.Random functions, using an overall seed value and one for each part of the game, e.g., bird song, bird appearance, island biomes etc. We save the seeds and also a count of how many values have been generated, allowing us to save the state of the game at any point and restore it later without having to store all of the potential values.
Ironically, using straight random numbers feels less organic than using standard deviations, so we used Unity’s Animation Curves to create bell curve distributions. The most straightforward way to do this is to create a curve like so:
and then generate random float values until you find one that is below the area of the curve, then multiply by a high and low range value. This method was used to determine values for everything from the appearance of the birds, to how often they would sing, to the values fed to the RTCP ramps in Wwise that controlled their songs.
We also used perlin noise samples–particularly in the terrain generation–to produce organic-feeling randomness. Perlin noise generates a smoothly varying set of values between 1 and 0, often represented by color values between 1 and 0 on a black and white texture. We then traverse the pixels on this texture and “sample” the values to give us a random number between 0 and 1. Due to the smooth gradient between values there is no sudden jump between two random values and can result in a seemingly organic result when used to model something like terrain, clouds, or the variation of a shoreline. Something we are interested in doing in the future is combining this with the use of fractals to provide an even more organic set of random outputs which more closely resembles the ordered randomness found in nature.
