Wow I can’t believe it’s been 2 weeks since the last post. Basically I went down a giant rabbit hole of balancing…

It all started because I wanted to give a smuggler early on in the game some Slasher’s Cider, which is a potion that increases the chance to cause bleeding. However, when I ran my automated AI test system the potion didn’t seem to be make much of difference to the outcome of the fight.

So I embarked upon a large scale test of all the bombs and potions in the game to check that none were overpowered or underpowered. This involved setting up special test scenarios for each item and logging the results.

How many tests to be accurate?

Typically I run the automated AI test system 500 times and then again and compare the results. I’ve found that running a fight just 100 times isn’t accurate enough.

In fact I was reading about the margin of error for repeated tests and for 10 tests it’s about 30%, and for 100 tests it’s about 10%. That means I could run a test and get say a result of 40% and then run it again and get a result of 60%, or anywhere between those values (if the true value is 50%). That’s not acceptable for fine tuning the effectiveness of bombs and potions, or anything really!

Running 500 tests reduces the margin of error to 4.5%, which isn’t great but is better. Plus if I run it again and average the two values, it becomes even more accurate.

What did I discover?

Early on the the design of the game I gave the various bombs and potions placeholder values and ordered them in what I thought was a reasonable order of effectiveness. However, my tests soon revealed that quite a lot of them were underpowered (nothing was super-overpowered but there are a couple of powerful items on purpose).

So I embarked upon various tweaks and retesting until I got all the items to influence the outcome of fights in an acceptable manner.

It’s possible players might have used an underpowered item but not really noticed it was underpowered just because it did something visually cool and changed a stat, but I didn’t want to take that chance.

250,000 tests!

In the end I ran about 250 test scenarios and many of those were run twice to collect more data. Each of those scenarios consisted of 500 automated AI tests. So in total that’s about 250,000 tests.

Imagine if I’d tried to do that manually? Or just said “those figures are probably good enough”. The game could be broken.

Some may call me perfectionist (and they may have a point) but I learned a lot about how the potions influence the outcome of fights and if they are best used with fast or slow weapons, one or two weapons, at the start of end of a fight etc. I could probably write a Shadowhand strategy guide now 🙂

Anyway, I’m back on track with the main development now, so more blog posts coming soon!

This entry was posted on Tuesday, August 15th, 2017 at 4:56 pm and is filed under Developer Diary, Game Development. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.