Evaluation

[update]
I’ve since completed this project and finished my internship at Little Chicken and graduated from school with a Bachelor of ICT and Game Design and Technology degree.
[/update]

Since the last update I have mainly been working on side projects. Recently my colleague finally had time to do an evaluation of my persistence layer project. The results are very satisfying. There were only a few comments on issues meriting further consideration.

Automatic identifiers

Currently, every instance of an entity needs to be assigned a unique identifier to be persisted correctly. This is cumbersome to manage for the game developer and it might lead to mistakes and confusion. It was suggested to change this so that entities might be assigned a unique identifier automatically by the persistence layer.

This was something I did not consider at first, because I wanted the flexibility for the developer to be able to choose his own identifiers, but the idea is sound. My colleague pointed me to the GetInstanceID method on components which is ideal for this sort of thing and which I had previously overlooked somehow. There is only a trivial change required to implement this change.

Hierarchy

With GetInstanceID it should also be possible to distinguish different Transform components, making persisting and restoring parts of the hierarchy an option again.

Unfortunately all the other issues with the hierarchy that I noticed before this have not disappeared. Restoring the hierarchy will still be a very complex issue, with a lot of potential for bugs and edge-cases that will be difficult to deal with, making the overall system less reliable.

I still feel that this feature is more of a ‘nice to have’, than something essential, so for now I will focus on polishing the rough edges, writing documentation, and the finishing touches on my thesis for school.

Multiple Entity Components on a spawned GameObject

This was a question which confused me a bit. Why would you ever put more than one entity component on the same prefab? This way they can not be separated so they are effectively the same entity, which defeats the point of having multiple entity components. I could only conclude that it was a desire to organize things.

Due to the way this system is designed, dealing with this strange edge case is not possible, as the system would try to spawn two game objects from prefab while there should actually be only one.

Conclusion

While there are still a few things to be desired, the overall system works fine and fulfills its purpose. All that remains is to polish some rough edges and write documentation. I will also put the finishing touches on my thesis for school, which is nearing the deadline. After that, I might still have time to look at handling the hierarchy.

Analysing the second prototype

It’s been a while since the last update. The last month I have been working more on the main project, which successfully passed an important deadline. The past weeks I have been working on my own research and assignment. The first prototype has been well received and with this second prototype, the project is slowly nearing completion.

The past few weeks I have been expanding on this to create the second prototype, or iteration, for my Unity persistence layer. I detailed the major weaknesses of the first prototype in my previous post, along with some other considerations. The most important problems were solved in the second prototype, but there were some difficulties as well and some problems turned out to be impossible to solve reliably.

Chosen solutions

Dynamic Entities Instantiation

The biggest weakness of the original prototype was the fact that it was only able to fill existing entities with persistent data from storage and not able to instantiate new entities in the scene if this was required for the game. Initially, a work around was used to mitigate this problem.

In the new prototype, this problem has been solved by the addition of the Type property and the IEntityFactory interface. Developers using this persistence layer can implement an entity factory and plug it into the system to add enable instantiating dynamic entities.

This solution was chosen because, depending on the game and the design and implementation of game entities, instantiating an entity is likely to be quite complex. In Unity, game entities are likely to be inherited from the MonoBehaviour class, which means that they can not be instantiated normally. Instead, they usually need to be instantiated from a prefab, together with meshes, colliders and other components they are composed of.

The type property allows the EntityFactory to select the right prefab, even though the entity classes of different prefabs might be the same. For example, a red car and a blue car might each have their own prefab, but both are a Car entity. Note that entities do not HAVE to be inherited from MonoBehaviour and can also be plain old c# objects. In that case the factory can just ‘new’ an object like normal.

Handling References between Entities

With dynamic entity instantiation, there occurs a related problem, namely the handling of references between entities. Since entities are instantiated elsewhere, we would need to override the serialization behavior to return references to these already existing entities. This is a very useful feature to have when dealing with collections of entities.

An appropriate example would be a Track entity which maintains a list of all TrackPiece entities.

This presented a bit of a problem because these references to entities might be embedded quite deep in the persistent data fields of an entity. This means that the serialization system needs to deal with this somehow. Whatever serialization mechanism is used needs a way to detect the presence of a reference to a game entity, and instead of trying to serialize it, must write the id of this entity, because the entity itself will be handled manually by the rest of the persistence layer. Upon loading, the serialization mechanism will again need to recognize that it should not attempt to deserialize references to entities in the normal way, but instead call back up to the persistence layer to obtain a reference to the entity with that id.

This was accomplished two of the three implemented encoders. In the JsonEncoder the JsonConverter class could be overridden to change the behaviour of the JsonSerializer.

Likewise, in the BinaryEncoder, the ISerializationSurrogate interface was used in a similiar way.

Unfortunately the XmlSerializer used by XmlEncoder did not support any feature like this and the more modern XmlObjectSerializer is not included in the version of Mono that ships with Unity, so this encoder does not support serialization of references between entities.

Restoring the hierarchy

With dynamic entity instantiation, another problem that crops up is the scene hierarchy. In an variety of situations, it might be needed to restore an entity to its proper location in the hierarchy.

This feature was attempted, but abandoned due to several reasons.

Writing the hierarchy data did not prove to be a significant problem. Recursion was used to filter the transforms of all entities and their parents. These were written to the save file and loaded again correctly.

Unfortunately restoring the hierarchy in the scene properly proved impossible to do because of limitations in Unity. This stems from the fact that individual transforms in the scene do not have any unique identifiers. Different transforms can have the same names which leads to unpredictable behavior when loading since different transforms can not be distinguished from each other.

The only way to implement this would be to assign a unique identifier to each transform upon saving and upon loading to load every single one back into the scene. Obviously this does not work in cases where some entities are already defined in the scene editor as they would be duplicated instead of merely filled with the right data. This solution would make saving and loading an all-or-nothing approach and does not fit with Unity’s design philosophy in my opinion.

Another, less important reason to scrap this feature was that saving the entire hierarchy proved to save quite a bit of redundant data that might not be needed. Instead, having each entity manage it’s own location in the hierarchy if relevant was considered a better option. This option is less friendly to the developer but offers a lot more flexibility to the developer, who can make his own judgment on how to handle this issue on a case by case basis.

Integration

To enable easier integration with other systems, a change in the design was necessary to make the method of storage more accessible, so that parameters can more easily be changed and data can be exchanged. The new design is further described in the following section.

Design

The design of the system was changed quite a bit compared to the initial design.

The IPersister and IPersister manager and implementing classes have been merged. This makes it easier to access the storage options or data from outside.

In addition, there is now an abstract base class StreamEncoder, which handles extracting the data from entities and instantiating them. Subclasses inheriting from StreamEncoder override abstract methods to implement serialization details and writing and reading all data to and from the stream. This separates the actual extraction and restoration work from the serialization method chosen, even though they are still tightly related.

The additional feature of handling references between objects required a change in the general algorithm for loading entities and restoring the scene. This now happens in two passes, first instantiating all entities, and then filling all entities with data and reconnecting references between entities.

Evaluation

Since the original Workshop project is pretty much complete right now, testing these changes is better done somewhere else. Having a smaller project also helps with testing only the functionality of this system.

To test and evaluate this prototype, a new project was created containing a test scene with a collection of entities implemented to cover all functionality during testing. The following features were tested in the following ways:

• Persist static entities. Saving and restoring entities defined in the scene in the editor. Three moving Ball entities were placed into the scene and given persistent properties for position, orientation, scale and velocity.
• Persist dynamic entities. Saving and restoring entities instantiated at run-time. A BallSpawner entity was added to the scene which instantiates smaller ball entities on the press of a button.
• References. Saving and restoring persistent references between entities. The BallSpawner was given a list with references to all balls it spawned. Balls were given a reference to the last Ball they collided with. All of these references were rendered by drawing lines in the scene for easy visual confirmation.

Together this test scene and it’s contents covered all features I designed into this system. Static Balls are correctly restored to the right position on load. Spawned balls are instantiated correctly and restored to the the right position as well. All references among entities are restored correctly.

Strengths

All strengths of the first prototype, plus the following:

• Support for dynamic entities
• Support for persistent references between entities
• A more flexible design with less code duplication

Weaknesses

• The game object hierarchy is not restored automatically. In situations where this is needed, the developer will need to take care of that himself, usually by implementing entities’ OnAfterLoad methods.
  Note that transform members like position can still be easily persisted by wrapping them into a property. Restoring the hierarchy could be as simple as attaching the entity’s transform to a parent entity.
• Some serialization mechanisms can not support references between entities and other features like polymorphism. At the moment this is the case with the XmlEncoder.
• This will need to be considered. Special care still needs to be taken to ensure the value types of persistent fields and properties are serializable in the chosen serialization mechanism. Every serializer demands it’s own rules and constraints for serializable types. This really can not be avoided.

Other Comments

• Recently another use case was discovered which, in hindsight, should have been considered. That is the case where game entities are defined in the scene and removed during run-time. An example of this would be pickups that vanish when the player touches them. This should be handled in the final version but is not a difficult problem.

Conclusion

The current system is usable for most of the use cases it is likely going to deal with. One exception is the use case for removed entities, which was not initially considered, but handling this should not be difficult so it will likely work in the final version. Overall the project seems on track, including my thesis which is slowly accumulating a nice amount of content.

Analysing the first prototype

The past weeks I have been working on working on some general gameplay functionality for the main project, in addition to further integration of the first prototype of my persistence layer. It is now possible to store the game state in a save slot on a server. I have also analyzed the current strengths and weaknesses of the system. The following diagram shows how it is designed:

The most important parts are the IEntity interface which is implemented by game entities and enables them to mark fields to be persisted and to receive signals after loading and other persistence events. The bulk of the work is handled in the implementations of IStreamEncoder, of which currently only XmlEncoder is implemented and uses reflection and the XmlWriter and XmlSerializer classes to serialize the entities and their fields to an xml document. Which can be saved to a file, or somewhere else, by the IPersister.

I described the relevant problems and my chosen solutions in my previous post so I will not go into those here and now. I was mostly on the right track, and the overall system is working fine, though there are still quite a few points of improvement.

Strengths

• Seperation of responsibility: Serialization and storage details are abstracted away from the rest of the game logic.
• Simple and Intuitive: Adding extra persistent game entities is easily done by implementing IEntity and marking the relevant fields and properties with the PersistField attribute.
• Convenient: It is possible to persist fields defined in an object related or attach to the entity by wrapping them in a persistent property.
• Flexible: different implementations of IPersister can easily be switched out to change the overall behaviour of the system.

Weaknesses

• Dynamic entities: The current system collects all entities in the scene during saving, and extracts their data. Upon loading, all entities in the scene are again collected, and filled with the data that was saved earlier. This works for scenarios where the state of entities changes, but it does not work in scenarios where entities have to be created, or removed to reach the saved state. We used a workaround in the form of a manager entity to handle the state of a variable number of other game elements. Recreating entities at the right place in the scene and hierarchy would be desirable. This also will require a way to instantiate game entities without needing references to prefabs.
• Dependencies: Some entities turned out to have dependencies with others and expected those to have been initialized already. I quickly solved this by adding a priority to IEntity to govern the order in which OnAfterLoad is called on them. This is perhaps not the best solution. Priority based on location in the scene hierarchy might be an intuitive way to handle this, but is less flexible. Food for thought.
• Integration: Integrating with existing systems turned out to be a bit trickier than expected. The save methods do not take parameters so options like save names needed to be set in the current IPersister while ideally, only the IPersistenceManager should be accessed once the system is configured. This part of the design could be reworked.

Other Comments

• IEntity: This interface now contains four methods implemented by entities to receive signals when needed on persistence events. These should be optional but the interface requires an implementation, which can be irritating. This could be changed.
• Design: As of this moment the bulk of the work happens in the XmlEncoder, which will require duplicating functionality when different encoders are desired. The extraction of data and serialization of data should be separated.
• IEntity: My colleague offered another suggestion to handle the extraction problem. Instead of code annotations, this would involve an object with a list of names of fields to be persisted, which could be attached to a game entity. In Unity this could be a component, granting the user the ability to set values to save in the inspector. It is an interesting idea which merits further investigation.

Conclusion

The current prototype is quite successful. It does the job and does it reliably, and has kept the persistence implementation of most game entities relatively simple. It has proven that the chosen solutions work for this problem. There are still aspects where it falls short, which should be addressed in the following iterations of new prototypes. Handling dynamic game entities and dependencies should have the highest priority right now.

The First Prototype

As planned, I have been working on my first prototype for my persistence layer. This prototype will be integrated into a current project with a deadline so it needs to be simple.

I have chosen the most promising methods for dealing with each sub-problem which I have identified in my earlier preliminary research.

Extraction

By extraction I mean the method by which the relevant data is collected and extracted from the current state of the game. I found three candidate solutions to solving this:

1. Domain model: Have all game objects contain references to separate data objects.
2. Reflection: Have all game objects use annotations to mark data they want serialized, and extract using reflection.
3. Manifest: Keep all data separate in a single object graph, commonly called a manifest

I decided that reflection is the most promising candidate for the first prototype, since it would enable a lot of flexibility and not require a major restructuring of the already existing code base. It results in a slightly more complicated persistence layer but is offset by the ease of development for the user. It was inspired by the way object relational mapping is commonly used in Java and to a lesser extend .NET systems.

The domain model and manifest solutions would result in a simpler persistence layer, but would require a lot of changes in the existing code base to supply the relevant data.

Serialization

I am working in Unity, which means I have access to most functionality in .NET implemented by Mono, including serialization.

1. XmlWriter/XmlSerializer: serialize to XML
2. BinaryFormatter: serialize to binary stream

The choice of serialization method to use is sometimes very dependent on the situation, but usually comes down to preference. In this case I was given to understand that the data would be sent to a server later, so encoding to text was recommended. Xml was the logical solution to use in this case.

Storage

During testing, storing data on the filesystem makes analyzing it and debugging easier, but later the data should be sent to a web server connected to a database. I decided to make the system flexible by creating interchangeable modules to change the behavior when needed. I made one module to save to the filesystem using a FileStream and another to interface with a different system handling the remote server.

Reconstruction

To prepare for reconstruction, I made sure to structure the serialized data in such a way as to be able to keep it apart later and still know where everything belongs. This is why I gave every entity a unique identifier. This allows the system to restore all the data to where it belongs. The actual restoring of the scene can by a combination of the following methods:

1. Automatically save and restore entire scene hierarchy related to the entity
2. Save entity and use custom methods per entity for restoration after loading the data.
3. For dynamic entities that should be instantiated, the factory pattern is a good approach.

I decided to go for the second option for two reasons. The most important reason is that option one would make the persistence layer very complex, and I do not have the time right now to implement such a solution, while the sacrifice in usability is very small. Each entity can easily handle its own initialization after being loaded with data by implementing OnAfterLoad and other methods. The other reason is that the first option might lead to a lot of redundant data, which might create storage issues and would introduce a lot of places where things could go horribly wrong.

To summarize

I have chosen the best solutions for the sub-problems identified and started implementing the first prototype. Initial results seem very promising. The prototype has already been integrated into the main project to have actual data to save and it is performing nicely. There are still a few kinks to work out, and a few glaring shortcomings, but it will serve for the current purposes.

Research on Persistence

The past week I have been doing preliminary research on persistence in the context of game development in preparation to developing a standard system to handle this.

I have scoured the web and found quite a few sources on this topic, though most of them have not been very useful. Most questions and responses in the Unity community focus on the smaller problems like serialization. There is very little deeper discussion on the design of systems that should handle this. The same goes for most game development forums. It is commonly said that every game has different needs and thus persistence should be custom-built for every game. I am not satisfied with that, because I feel that most of them deal with fundamentally the same problems, just in different situations. In my opinion, a lot of this could be generalized.

The problem of persistence has been largely solved in enterprise software. There are advanced databases and other data stores available, with abstraction layers to ease development. Application functionality is divided into domain models for persistence and other parts for the rest of the functionality. This makes persistence a trivial problem for most software engineers.

This is usually not the case in game development. In my experience a lot of games are developed more in an ad-hock fashion than most  applications. For reasons of performance and ease of use, the data we want to save is often embedded in the scene in ways that make it difficult to get it out if persistence was an afterthought. Any Unity developer who has ever tried to save and restore entire game object hierarchies at run-time will know the difficulties I am talking about. Whether these designs are wise is another matter I will not go into here. In addition to this, there is the fact that different games use different storage methods. Some might use a save file on a local hard disk while others will save state to a web service or database. All these circumstances make persistence for games a touchy subject.

I took another close look at all the resources I had gathered and noted that the entire problem of persistence in game design can be summarized in a few sub-problems:

1. Extraction: Separating the data you want to persist from the rest of the game
2. Serialization: Encoding the data into a sequential format suitable for storage and back
3. Storage: Actually storing and retrieving the serialized data
4. Reconstruction: Reconstructing the scene or game state from loaded data

In a lot of situations, these problems might overlap or even blur as to be almost indistinguishable from each other. At all times every one of these problems is present in some way. Most resources I have found focused on serialization and storage. The more difficult problems are actually extraction and reconstruction, but often overlooked, which often causes problems in serialization.

For each of these problems I have found a number of ways to deal with the problem. I will choose the most promising solutions I have found and integrate these into my first prototype.

Internship at Little Chicken

Soon after the jam I started on my last semester of my studies and my planned internship at Little Chicken Game Company in Amsterdam. Among other work, I will be doing research on and developing a standard, flexible, and easy to use system to handle persistence in games, in the Unity environment in particular.

Goals of this project:

1. Standard system: Take care of all the little details and make the game developer’s life easier when dealing with persistence in games.
2. Generic: Relevant and usable in as wide variety of use cases and types of games as practically possible, though there will always be exceptions.
3. Simplicity: Adding extra types of persistent game elements should be intuitive and involve minimal effort, and minimal intrusion into the rest of a project’s code base.
4. Flexibility: Easily adapted to situations requiring different forms of serialization or data storage.

The past weeks I have mostly been getting acquainted with the environment, setting up my workstation, and writing my project initiation document. I have also poked around at the existing code base to get familiar with the current project my solution will be integrated with.

I plan to do some preliminary research on my topic of persistence in games, after which I will develop a number of prototypes. For each prototype I will evaluate its strong points and shortcomings. This way I can iterate towards the final solution.