In this post we’ll be talking about some experiments we’ve been working with, to add mod scripting support for our future games. This is still in experimental stages, and we’re concentrating specifically on Unity, but we’re curious to see how it pans out. We’re also curious to hear if this kind of approach has been useful for others.
Fair warning: lots of geeky C# coding talk ahead. 🙂
Mods are a cool addition to games, and they can come in many flavors: art asset packs, skin packs, text files, etc. One very powerful type of a mod is a scripting mod, where modders write code that runs inside the game, and changes some behaviors at runtime depending on the mod logic.
There are three basic approaches to adding scripting to Unity games:
- Loading in executable code, native or .NET. This is traditionally difficult because modders don’t have good tools to do this. But – this is changing.
- Embedding an existing scripting language like Lua in the game. This is easy for modders, but a lot of work for the developers to support, and performance is considerably slower.
- Writing a custom scripting language and building an evaluator in the game. This is probably the most work, since it means creating a new language, parser, evaluator, etc., but it lets developers create a domain-specific language that’s tailored precisely to the needs of the game.
In this doc, we will discuss case #1. Unity games are written in .NET, which makes it relatively easy to load up external DLLs, but creating those DLLs is difficult for modders, since the tooling was historically difficult to set up and get started with. Games like Cities Skylines tried to improve on this, by distributing the entire tool chain right with the game, but this was a pretty difficult endeavor.
However, there is now an easier way. After Mono was re-released as an open source project, and after Unity 2017 switched to .NET 4.5, the new versions of Unity come with something called the compiler service. This is a full implementation of the Mono C# compiler, written in C#, and embedded into a self-contained DLL file called Mono.CSharp.dll.
It turns out that, with a little bit of work, we can embed this C# compiler right into a Unity project, such as a game, so that it will read C# source files at runtime and execute them.
This post will show how. We’re still experimenting with it, so I’d love to hear your thoughts on how well this is working for you. (EDIT: also please see security notes at the end!)
Suppose we have a game, for example an RPG, and that it includes a few standard character types. This will be our example test case.
Let’s say we now want to add scripting support, so that modders can add new characters, and code up new behavior for them, with their own code that executes when the game is running.
If we’re writing in C#, we might start with an interface that defines character types, and then create classes that implement this interface. Maybe the game ships with the following human class:Scripts/Characters.cs
public interface ICharacter
{
int Height { get; }
string Language { get; }
}
public class Human : ICharacter
{
public int Height { get { return UnityEngine.Random.Range(150, 200); } }
public string Language { get { return "Common"; } }
}
We can instantiate all subclasses of ICharacter at runtime like this, and list them:
var characters = wrapper.CreateInstancesOf<ICharacter>(); // explained later
foreach (var character in characters) {
Debug.Log($"Character of type {character.GetType()},
speaks {character.Language},
height = {character.Height} cm");
}
Which produces one console log entry like this, because we only have one built-in class that implements ICharacter:
 Character of type Human, speaks Common, height = 170 cm
Now we would like modders to create their own code and load it up into the game. Let’s say some modders are big fans of the middle earth, so they created a couple of fantasy characters as well, and they place it in a text file. For example, StreamingAssets/custom.txt (but the location doesn’t matter at this point):
public class Hobbit : ICharacter
{
public int Height { get { return UnityEngine.Random.Range(100, 140); } }
public string Language { get { return "Common"; } }
}
public class Elf : ICharacter
{
public int Height { get { return UnityEngine.Random.Range(170, 210); } }
public string Language { get { return "Elvish"; } }
}
Now the big question is: this mod code lives in a text file. It’s created by modders, so it doesn’t ship with the game – it needs to be loaded up after the game starts.
How do we compile it into .NET and load it into the game?
First, let’s get set up.
As mentioned in the introduction, Unity 2017.x comes bundled with a C# “compiler service”, which is a DLL file that includes the Mono C# compiler. This is a part of the Mono tool chain which has recently been re-released under under a broad open source license.
The compiler doesn’t get built into project automatically (it’s a 1MB+ code library!), but we can bring it over ourselves.
1. In “Build Settings”, switch the game API compatibility level to “.NET 4.6 (Experimental)”
2. Now let’s find the compiler DLL. For me it lives under C:\Program Files\Unity 2017.2.0p2\Editor\Data\MonoBleedingEdge\lib\mono\4.5\Mono.CSharp.dll
3. Copy this DLL over to your own project
If we play the project as is, it should start up without errors, but not actually do anything yet (try it!).
The second step is to initialize the Mono CSharp compiler. The bulk of the compilation work is done by the Mono.CSharp.Evaluator class, which needs to be set up.
We will do this by wrapping it in our own manager class that will do the setup work, and then limit what the compiler can do. The following example is based on the excellent prototype by Michael ‘Searge’ Stoyke:
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using System.Text;
using Mono.CSharp;
public class CompilerWrapper
{
private Evaluator _evaluator;
private CompilerContext _context;
private StringBuilder _report;
public int ErrorsCount { get { return _context.Report.Printer.ErrorsCount; } }
public int WarningsCount { get { return _context.Report.Printer.WarningsCount; } }
public string GetReport () { return _report.ToString(); }
public CompilerWrapper () {
// create new settings that will *not* load up all of standard lib by default
// see: https://github.com/mono/mono/blob/master/mcs/mcs/settings.cs
CompilerSettings settings = new CompilerSettings
{ LoadDefaultReferences = false, StdLib = false };
this._report = new StringBuilder();
this._context = new CompilerContext(settings,
new StreamReportPrinter(new StringWriter(_report)));
this._evaluator = new Evaluator(_context);
this._evaluator.ReferenceAssembly(Assembly.GetExecutingAssembly());
ImportAllowedTypes(BuiltInTypes, AdditionalTypes, QuestionableTypes);
}
This looks simple, except for the one trickiness about the compiler settings. By default, the compiler would import the core library and make it available to the modded scripts. We want to prevent this, in order to prevent modders from accessing dangerous types like System.Reflection.
The first step is to disable them in compiler settings, listed above:
CompilerSettings settings = new CompilerSettings
{ LoadDefaultReferences = false, StdLib = false };
Secondly, we create a function ImportAllowedTypes() which will explicitly import some types of our choosing into the Evaluator. This way we can control what types are available in mod scripts. For example, if we don’t import any types from the System.IO namespace, the compiler will not be able to compile code that uses those types to access the filesystem (but see notes at the end of this post).
Since we did not import the core library, we will need to import some of the basic standard types one by one, otherwise the script won’t be able to do much at all. But as a benefit, we’re free to only expose types that we think are safe for use by mod scripts.
Importing types is going to be a little tricky, because we need the Evaluator.importer and Evaluator.module fields, which are marked as private. However, as we may remember, private access modifiers are compiler-time restrictions but they do not provide runtime security. We get around them via reflection.
The following solution, based on Searge, gets an array of arrays of types, and imports them:
private void ImportAllowedTypes (params Type[][] allowedTypeArrays) {
// expose Evaluator.importer and Evaluator.module
var evtype = typeof(Evaluator);
var importer = (ReflectionImporter)evtype.GetField(
"importer", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(_evaluator);
var module = (ModuleContainer)evtype.GetField(
"module", BindingFlags.Instance | BindingFlags.NonPublic).GetValue(_evaluator);
// expose MetadataImporter.ImportTypes(Type[], RootNamespace, bool)
var importTypes = importer.GetType().GetMethod(
"ImportTypes", BindingFlags.Instance | BindingFlags.NonPublic, null, CallingConventions.Any,
new Type[] { typeof(Type[]), typeof(Namespace), typeof(bool) }, null);
foreach (Type[] types in allowedTypeArrays) {
importTypes.Invoke(importer, new object[] { types, module.GlobalRootNamespace, false });
}
}
Now we can use this to import a variety of different types, using this line:
ImportAllowedTypes(BuiltInTypes, AdditionalTypes, QuestionableTypes);
And here are the definitions of different types we allow modders to use:
/// Basic built-in system types.
private static Type[] BuiltInTypes = new Type[] {
typeof(void),
typeof(System.Type),
typeof(System.Object),
typeof(System.ValueType),
typeof(System.Array),
// ...
// etc.
// see full list here: https://gist.github.com/rzubek/e1b73e2262f56e04f9f979a8203bf0c7
};
// we also need to define
// private static Type[] AdditionalTypes
// private static Type[] QuestionableTypes
NOTE: for AdditionalTypes and QuestionableTypes, see the full source code here.
Finally, we add helper functions to round it out:
/// Loads user code. Returns true on successful evaluation, or false on errors.
public bool Execute (string path) {
_report.Length = 0;
var code = File.ReadAllText(path);
return _evaluator.Run(code);
}
/// Creates new instances of types that are children of the specified type.
public IEnumerable CreateInstancesOf<T> () {
var parent = typeof(T);
var assemblies = AppDomain.CurrentDomain.GetAssemblies();
var types = assemblies.SelectMany(assembly => {
return assembly.GetTypes().Where(type => {
return !(type.IsAbstract || type.IsInterface) && parent.IsAssignableFrom(type);
});
});
return types.Select(type => (T)Activator.CreateInstance(type));
}
The first one loads mod script file and evaluates it as a C# source file. The latter is a helper to create all instances of ICharacter, so that we can test whether user code got loaded up.
We have created a couple of modded classes in StreamingAssets/custom.txt, and we have added the CompilerWrapper to our project. Now we’re ready to test our mod.
Let’s write a little test harness:
using System.IO;
using UnityEngine;
public class ScriptTest : MonoBehaviour {
void Start () {
var wrapper = new CompilerWrapper();
// load text files and run them
foreach (var file in Directory.GetFiles(Application.streamingAssetsPath, "*.txt")) {
wrapper.Execute(file);
Debug.Log($"Read file {Path.GetFileName(file)},
errors: {wrapper.ErrorsCount},
result: {wrapper.GetReport()}");
}
// see what we got! this includes built-ins as well as loaded ones
var characters = wrapper.CreateInstancesOf<ICharacter>();
foreach (var character in characters) {
Debug.Log($"Character of type {character.GetType()},
speaks {character.Language},
height = {character.Height} cm");
}
}
}
Don’t forget to attach this component to some object in the scene! Now if we run it, we should see this in the console logs:
Read file custom.txt, errors: 0, result:
Character of type Human, speaks Common, height = 171 cm
Character of type Hobbit, speaks Common, height = 136 cm
Character of type Elf, speaks Elvish, height = 200 cm
This shows that the game correctly loaded up the mod file, and now recognizes three classes that implement ICharacter: the built-in Human type, and two types loaded up from the mod file.
Finally it’s time for some important caveats.
First, since the Mono C# compiler performs compilation at runtime, it will not work with IL2CPP / AOT targets, including iOS. It requires runtime .NET support for System.Reflection.Emit, which probably means it will remain restricted to desktop platforms.
Secondly, a couple of notes on security (thanks to SeargeDP for extra info!).
A. In this version of the wrapper, mod scripts are given access to all types in the assembly that contains this wrapper – that is, all types in the game. This makes interoperability with the game easy, but also might expose a variety of internal types that are not meant for outside consumption, and could be security risks. To restrict compile-time access, it is best to:
- Remove this line:
this._evaluator.ReferenceAssembly(Assembly.GetExecutingAssembly()); - Manually list all game types that the script can access, inside
ImportAllowedTypes
B. Mod scripts should be sandboxed as much as possible, so that they can’t perform unsafe operations. Ultimately, it would be best if the entire CompilerWrapper was run in a standalone sandbox AppDomain. But the cost of doing that is high – we can’t share types between app domains, they have to communicate via serialized objects, etc.
So in this case we load the scripts into the same app domain as the game, and try to sandbox it as well as we can. The lists of allowed types are the first line of defense: as long as the list of allowed types is limited to only safe ones, and System.Reflection, System.IO and other such are not allowed, this will remove the obvious avenues of misuse.
However, there are other reflection-based leaks in the sandbox. The System.Type class must be imported because it’s fundamental, but it also exposes a variety of reflection methods. Also, dynamic elements (see dynamic keyword, Linq expression trees, etc) must be excluded since they can also be used to bypass compile-time type checking.
We may wish to exclude expressions such as dynamic, typeof, or methods like GetType, GetMethod, GetField, etc. from the source files that we’re loading. Ideally this would happen as part of loading up and parsing those source files. However, that’s material for a future research post.
And that’s it for this post. If you’re using something like this already, or have thoughts on this, please get in touch, I’d love to hear from you! @rzubek on Twitter.
