I am starting a new open science experiment! I want to describe here on my blog (but also on my Twitter) all the steps I am taking on the small research project of trying to make the game The Sims4 accessible to visually impaired gamers. Come and let’s research together!

I don’t know how this thread will end and when. Maybe tomorrow I will say « sorry I cannot pursue this project because X Y and Z », but that would already be a research output! The goal here is to share how difficult it is to make an existing game more accessible from a researcher and developer point of view. I will start with some desk research, do technical research and maybe more… Maybe at some point you will have information that could help me making The Sims 4 accessible. And, if you share it, it will be participatory science!

Maybe I will start with a small presentation of my PhD. I am studying to what extent could generated audio description be used to make gaming accessible to visually impaired people while ensuring a balance between information and entertainment. My PhD also focusses on live streaming. I already made some technical prototypes, did theoretical research, but there is still a lot to do. If you want to learn more, there is this presentation: dav.li/TFTIPGS21/! I identified four main ways to get game data in order to generate audio description in real time: send direct requests to a server from the game, use telemetry SDKs already embedded into games, retrieve data with public game APIs (for eSport games), or mod an existing game. For the two first ways, you need to have the game developers involved and the third way is only available for eSports games. I wanted to dig more the last way: modding an existing game. Modding it when a player modify a game with some code.I am a player, so let’s try to do it!

First, we need to find a game to mod. I have already done this part as I told that The Sims 4 will be the game I will try to mod. I chose it because I like the game: I played to all Sims games! Good news! The game is moddable (we can do mods). We can find on the web a lot of websites about mods for The Sims4: sims4modding.com for instance. There is a lot of mods that adds objects or modify the gameplay, but I couldn’t find mods that improve accessibility. I found this post on Reddit where someone is looking for mods that improve the accessibility of the game for a visually impaired player. I have also found this forum post from a gamer that is loosing sight and want to continue to play The Sims 4. On Reddit, a gamer is asking « How accessible is the Sims for visually impaired people? ». That’s a good question to ask before coding a mod! On Taming Gaming DB, the accessibility report states some accessibility features (like scaling the UI). A quick search on Can I Play That output no result for any The Sims games. The Sims 4 is not on the EA accessibility ressource page.

Technical side

Now let’s research the technical side of modding the Sims 4! We can find some official documentation for modding: Maxis info index, Maxis documentation. We can also find community documentation and tools: List of TS4 Resources, How to Access The Sims 4 Python Files and a community library for modding the game! This mean that we can mod the Sims 4 not only to change its look and feel, but we can also modify the gameplay using Python scripts. For instance, the official documentation provides an example mod called « maslow » which fills all commodities of all Sims on the current lot. This also mean that it seems technically possible (and easy) to code a script that describe objects present in the game. Let’s install the Sims 4 and test the example mod first!

I have achieved to install the example mod. It outputs text when triggered, but my Sim is still hungry. Maybe the mod is using outdated functions. I tried to use a mod library made by the community and install the example project. It is showing the notification when loading a household as expected! I think I will use this tool to build my mod. This tool is well-documented (read: sims4communitylibrary.readthedocs.io) and is in active development (last update was 2 days ago).

I have just achieved to create a simple Sims 4 mod that is requesting some text from a web server! This means I can reach my web API. As I understood, the Sims 4 modding environement do not permit to install libraries as you want. If you have the source code of a library, and all its dependancies, you technically can install it by putting the source files in your mod, but it’s complicated. The Sims 4 has Python 3.7 embedded with some libraries like the HTTP module. So it is possible to make HTTP requests, but not HTTPS ones as SSL is not installed. I also wanted to use a text-to-speech module like pyttsx3, but installing its source code with all its dependencies seems not feasible at this point.

My mod is now sending data to my web API. It sends the action name, the target object name and the character name. It also indicates if the author of the action is a playable character.

{
  interaction: 'John McArthur running cake_Grab:41 on object_Food_PlateFork_FruitCake_Multi:0x05c90dcf043015d0  (guaranteed)',
  action: 'cake_Grab',
  target: 'object_Food_PlateFork_FruitCake_Multi:0x05c90dcf043015d0',
  character: 'John McArthur',
  npc: 'False'
}

This is great, but the data is not really human readable. This will be the next technical challenge: create meaningful descriptions!

I managed to connect to my API and transform some data to description. Not all of them are transform, but here is a small demo video:

I have also published it on Reddit and on the GameToSpeech website.

This was an example where I hard-coded some values, but we need to go industrial. To do so, I have extracted some data directly from the game files. After some coding, I achieve to retrieve 7746 labels from the internal game files! They will be very useful to create meaningful descriptions. It will be easy to transform objects ids into human-readable text.

The mod has been released and it is now in closed beta! You can learn more on the GameToSpeech website or directly on the repository.