Dissecting SEGA's Shenmue Patent: Magic Weather & Magic Rooms

In today's post we will examine a patent application submitted by Sega back in 1998 relating to the generation of graphics suitable for interactive role playing games (RPGs), with the inventors listed as Yu Suzuki, Takeshi Hirai, Kazuo Kondo and Kazunori Masuda.

While Shenmue itself is not mentioned by name within the document, of course, the concepts described within can be readily recognized as pertaining to the soon-to-be-published game series. This includes two algorithms in particular for the automated generation of in-game content that were designed to save having to store vast amount of data on the limited space afforded by physical discs: Yu Suzuki dubbed these Magic Weather and Magic Rooms. We will focus on these two systems in this post.

The patent document contains 24 pages in total, over half of them being diagrams and flowcharts. 

The US patent document contains 24 pages

As a whole, the patent brings together a number of concepts that describe what we know today simply as the "game engine". The motivation was to ensure that the Graphical Processing Unit (GPU) receives display data fast enough to ensure that the display is updated frequently enough to maintain smooth output. The game prepares and sends a single image, or "frame", to the GPU in a never-ending loop. This means that the game's logic processing must be handled efficiently in the time period between one frame being sent to the GPU to be displayed, and the next.

An English-language version is available in the form of the US patent application document, from which the diagrams in this post have been sourced. The language used in the English version is rather stilted and confusing, so relevant snippets of text have been translated from the Japanese original in preference. Additional color images have been included for illustrative purposes.

Before diving into the patent itself, let's have a brief recap on what these two systems are about.

Thanks to LemonHaze who provided technical explanations and information about the implementation in the first two Shenmue games.

Magic Weather

Revolutionary when featured in Shenmue, the "Magic Weather" system controls the ever-changing climate and weather conditions experienced during play.

In the "Making Of" video documentary about Shenmue produced by NHK in 1999, the narrator introduced it as follows:

"The game contains a system that realistically recreates the weather and other natural phenomena. That system is called Magic Weather. In the game, the weather changes as time passes: fine, cloudy, rainy. These changes are based on the long-term weather data for Yokosuka, the town in which the game is set.

"Let's consider a case in the actual game where it is, say, fine one day and rainy the next. The weather conditions encountered by the player will differ depending on how they go through the game. For a player that moves ahead rapidly, they may visit a certain house on a fine morning. On the other hand, a player who plays more slowly may end up visiting the house the following night in the rain."

 

Magic Rooms

The second system, "Magic Rooms", refers to the use of a set of rules to automatically decide the layout of furniture across a large number of rooms. Its use in Shenmue II can be famously seen in the rooms of the buildings in Kowloon.

Yu Suzuki once described Magic Rooms as follows:

"When we were making Kowloon Walled City, I wanted to put everything in the rooms we made. There were thousands of rooms, and I wanted all of them to be free to roam. The designer told me it was impossible, so I thought there was no use in entrusting that task to him (laughs), and that’s when I thought, we could just make the program create them itself."

Next, we will move on to the patent itself, starting with details of the revolutionary new weather system.

Patent Application Details on "Weather Processing" (Magic Weather)

In the introductory section of the patent application, it is pointed out that one existing way to provide the player with a sense of the passing of time within an RPG by changing the color and appearance of the sky according to the time of day. However, it notes this does not fully represent changes in a natural environment, and that a better method is needed to portray it realistically.

Objective

To this end, the goal was to "provide games with improved reality and ambience by representing temporal changes in the outdoor natural environment in greater detail.".

The new invention simulates weather much more realistically, and describes the high level of control over various weather components:

"An image generator for generating images that incorporate a virtual weather environment inside a virtual three-dimensional space, with a means to identify at regular intervals the current camera position in the three-dimensional space and the current time; and a means for controlling the sky, weather, and weather intensity that make up at least part of the background on the display screen, corresponding to the camera position and current time identified."

In other words, it is a system that delivers changing weather effects according to the current time and location: a description that perfectly fits Shenmue's Magic Weather.

Details

An explanation is given of the weather processing, with an accompanying flowchart (Figure 1).

 Whenever the prescribed amount of time (for example, one hour) has passed since the previous weather processing, the weather condition is determined from a table of weather data, based on the current camera position and time.

Figure 1: weather processing flowchart

The weather table is described as having entries for each designated play area or location in the game.  It contains detailed settings including the date, hour of the day, appearance of the background sky, weather condition (for example: cloudy / foggy / rain / sleet / snow / clear) and intensity of the weather condition.

An example weather table is given in Figure 2:

Figure 2: example weather data table (date, hour, sky/background, weather, intensity).

The text goes on to explain: 

"Sunny and cloudy conditions are represented by adjusting the brightness of the entire screen, and this brightness is further changed according to the time of day. Wind (including direction and force) may be added to this weather information".

With the fine level of control that is possible, it might be expected that this would require a great deal of data to be stored, but this was not the case. The weather data could be stored in a compact format, with each entry in the weather table able to be expressed as a single 16-bit number (a range from 0x0000 to 0xFFFF in hexadecimal) as shown in Figure 3:

Figure 3: example weather data format

Despite its compact form, each entry would be able to specify one of 16 different sky images, one of 16 weather effects, and one of 256 intensity levels.

As the player moves between areas, weather conditions may change. The patent suggests handling this either by setting the weather condition to remain the same within each individual play area; or alternatively carrying out interpolation to ensure a smooth transitioning of weather condition between areas.

This smooth transitioning is implemented with combinations of "B-spline" curves (examples of which are illustrated in the diagram below), with each representing one of the weather types that exist. This means that, for example, snow can turn smoothly to rain - or any other weather transition imaginable.

Combinations of B-spline curves can be used to ensure smooth transitions

Implementation in Shenmue I

In Shenmue I, weather condition changes are defined within a date range of 1 November 1986 to 16 April 1987 (by comparison, the game's story commences on 29 November 1986 with the latest ending date being that of the "bad ending" on 15 April).

Famously, the game includes historical data for the actual 1986/87 weather in Yokosuka, although this can be enabled only after clearing the game; the first play-through uses a separate defined set of weather conditions.

Yu Suzuki highlights the weather system at the Shenmue Premiere in Yokohama

The weather settings data are separated into four weather types: Snow, Rain, Cloudy and Overcast; Sunny conditions are defined when all have zero intensity. Being compact in size, the game simply loads the entire weather table and holds it in memory.

Below is a small snippet of actual data for Snow, courtesy of LemonHaze, who notes that Shenmue I is one of the few games in existence where weather conditions are fully specified through a lookup table rather than just setting the effects when needed, or setting them randomly.

As can be seen, the data table follows a similar format to that laid out in the patent. (The "ValueRaw" is a number between 0 and 255, although this is converted to a fractional representation when used by Shenmue's code).

Day,Month,Year,Hour,Minute,ValueRaw
29,11,1986,8,0,255
29,11,1986,13,30,95
29,11,1986,14,0,0
7,12,1986,20,0,0
7,12,1986,20,30,159
7,12,1986,21,30,255
7,12,1986,23,0,159
7,12,1986,23,30,0
10,12,1986,22,30,0
10,12,1986,23,0,95
11,12,1986,6,30,95

The weather intensity value not only affects the graphical display, but - as confirmed by LemonHaze's code analysis - the code even has logic to adjust the audio volume for the weather-based sound effects based on the various intensities!

One of the features that was highlighted prior to the release of the game is the option of experiencing actual historical Yokosuka weather conditions, something which becomes available after the game has been cleared. As can be seen, this was able to be achieved simply by having a second set of alternative weather data.

Shenmue I has several background sky textures used for the "sky box" which vary according to the time of day and general weather condition. Here are a few of these:

Some of the background sky textures from Shenmue I

Implementation in Shenmue II

With Shenmue II, the fully-defined weather table approach is simplified by defining only the upper limits of the weather over a given time span; Pseudo Random Number Generation ("PRNG") is then applied, with changes smoothed as previously using B-spline curves, to determine the weather at any point in time.

This randomized approach is adequate, due to the limited number of different weather types; the game also does not support an option to replay using historical weather, which removes the need to support a full lookup table. The weather can also be set as required for specific events and cut scenes.

Rainy weather conditions (Shenmue II)

Benefits

The discussion of the Magic Weather system concludes by emphasizing the improvement over the other, simpler weather systems to date:

"Therefore, in comparison to conventional representations of weather conditions which merely change the sky (background) after a prescribed time elapses, it is possible to display detailed weather conditions realistically. This provides a sufficient sense of realism and increases interest in each respective scene of the RPG."

Details from the Patent Application: "Room Creation" (Magic Rooms)

The second invention relates to a new method to easily create varied rooms (partitioned space) inside a building, each filled with furniture and other objects laid out in a natural manner and without duplication. This is something that is often required when providing an RPG game environment.

Multiple entrances to rooms inside a building in Kowloon (Shenmue II)

The introduction text of the patent document notes that, conventionally, the creation of such rooms has imposed a vast processing load on the game, especially for large map areas. To reduce this load, measures are typically taken such as preventing the player from entering rooms where no events take place, or by cloning the layout of rooms. Such approaches are not ideal, however, as they feel unnatural and may contribute to the player losing interest in the game.

Furthermore, larger maps will result in high storage requirements and slower performance due to the quantity of data, especially on media with slow reading speeds, again leading to dissatisfaction for the player.

Objective

Accordingly, the objective was "to significantly reduce the operational load required for games that require rooms and maps, while at the same time taking advantage of the analog sensibilities possessed by humans in their creation and layout."

Yu Suzuki called the resulting process, employed for many of the rooms in Shenmue II's Kowloon buildings, Magic Rooms.

Details

To achieve a natural-looking layout for furniture and other items in a room, a room set-up routine in the game code is responsible for creating the rooms according to a set of special rules:

"This routine enforces rules for arranging furniture and other items based on human thought patterns and uses a random number table to arrange furniture and household items, thereby creating a room with a lived-in feel in the virtual game space."

A "Magic Rooms" furnished room interior, with windows in the far wall (Shenmue II)

As shown in the flowchart in Figure 4, firstly parts such as the walls and furniture are created (step S1).

Figure 4: room layout flowchart (corrected to add a step that was missing from the diagram in the English-language version of the patent)

Even the circumstances of the room's resident are taken into account (step S2). These include information such as "sex, age, whether he/she has a family, their family structure, health, financial status, and so on". A list of applicable furniture for the room is compiled, in accordance with these conditions (step S3).

"Next, the room is divided into areas, each of which has its own characteristics (step S4). An example is shown in Figure 5. In this example, areas 1, 3, 4, 6, 7 and 9 are the main areas where furniture is to be placed. For area 8, which is next to a window, the placement of furniture depends on the type of window - if it leads to a veranda then an object can not be placed here, but potentially a piece of furniture less than the height of the window may be placed. Area 5 is the central area for daily life and so a table may be placed there. Furniture would normally not be placed in area 2 to avoid blocking the door."

Figure 5: example of a room divided into areas (5a, left) and arrows indicating the directions in which furniture may face (5b, right)

The direction in which each piece of furniture faces is also decided. As shown in Figure 5b, possible directions are determined according to the area.

Next, the principle points of daily life in the room is determined (step S5, "Principle Elements of Living" in the diagram):

"The TV and audio equipment (especially the speakers), etc. are directed to this center point. In our example, the living focus is determined in the following order of priority: first, the center of the table, second, the center of the sofa (when there is no table), and third, the center of the room (when there is neither a table nor a sofa). As an exception, if the resident is fighting sickness, then the position of the pillow of the bed is set as the focal point for daily life."

Examples of furniture placed using the Magic Rooms algorithm in Shenmue II. See Jcgamer's indepth guest post series for a survey of the various items to be found in the Kowloon buildings.

Furniture is then arranged (step S6). Information pertaining to the arrangement of furniture includes:

• Furniture type: bed, cabinet, desk, table, TV, sofa, etc.
• Size: length, width, height of furniture
• Space required for use: length, width, height
• Storage space: (length, width, height size) x margin
• Possible placement areas within the room (e.g. area 1, area 3)
• Suitability for use: e.g. target age

The following layout rules will also be applied:

1. The layout area will be decided with a table of random numbers. By changing the random number seed, many different rooms may be created.
2. Pieces of furniture are placed in order of size starting with the largest, and those that do not fit are discarded.
3. Furniture that does not fit in a single area is placed by expanding into surrounding areas if they have empty space available. The surrounding areas will be reduced by the same amount.
4. If extra space remains in an area after placing a piece of furniture, the surrounding areas will be expanded accordingly.

Finally, everyday articles and objects for the walls are arranged (step S7). Foodstuffs, dishes, tableware, figurines, stationery, clothing, etc. are placed in areas allowed for each type. Wall clocks, paintings, posters, calendars, etc. are handled similarly.

Notes

The document summarizes the room creation routine by noting that it allows the developer to define areas (e.g., room corner, wall, center, ceiling, etc.) so that furniture will be placed naturally - not blocking windows or doors, and somewhere that suits the type of furniture. Parameters can be set up to indicate the characteristics of each room (e.g., tidy, cluttered, abandoned, warehouse, etc.), and the random number table can be adjusted to have such characteristics. By comparison, in the past, the approach has been to treat the entire room as a single object, so a great deal of time and effort was required to prepare many rooms.

Furniture is "placed naturally - not blocking windows or doors" (room interior from the Tea Break Bldg in Shenmue II)

Furthermore, it is noted, individual layouts have a human-like touch about them. For example, tables and chairs are less likely to be placed near a wall or in a corner of a room, where large furniture such as beds and shelves are placed. In addition, since the placement rules are not selected from a pattern, there is almost no chance that an identical room will be created, and a variety of rooms can be created in an extremely short period of time.

Benefits

The end of the document highlights the key benefit that this room creation method can provide: it substantially reduces the operation load of games requiring the preparation of room layouts and maps; and as a result, removes the need for the disk to be read every time a room is entered.

Final Comment

The patent document describes in detail two key innovative systems that will be readily recognizable to players of Shenmue, and to which Yu Suzuki has referred over the years. It is fascinating to examine Sega's own descriptions and gain insight into the way these systems work under the hood.

Acknowledgement & Thanks

Many thanks to LemonHaze, who discovered this patent's online existence and also explained several concepts and shared implementation details and data from his analysis of the actual Shenmue code, which have been incorporated into the above post.

Sources

Japanese patent application JP4240343B2 (filed 1998)

US patent application US7479958B1 (filed 1999)

The patents are marked as having been granted, but lapsed in 2018 and 2019 respectively.

Become a Patron!