Unity Nightmares - Alpha Cutoff

This is going to be my rant/nightmare about Unity with alpha cutoff shaders on mobile.

Probably the weirdest issue that I encountered in Unity had to do with opaque materials, that had the “alpha cutoff” flag enabled.

One of the scenes in my game consists of a rather empty plane, with a forest of trees surrounding it.

For some reason, when I got my hero to the edge of the plane towards one side of the trees, performance tanked.

What was even weirder is the fact, that it only happened on the south side of the plane. The other sides also had small performance dips, but the south was by far the worst one.

My initial hunch was, that the tree model that I used was too high poly. It was a model that seemed to have few polygons, but I suspected it just wasn’t optimized for mobile and had too much detail for me to plaster it around without performance issues.

So I asked an artist to create me “proper” low poly models. And he did - he pretty much halved the polygon count of the trees.

And the issue got worse.

I was completely demotivated and had to step away from developing for a couple of reasons, but this issue kept me from making a proper return to more important tasks.

That’s when I attended a talk about performance in games. And even though the talk itself didn’t give too much insight into this specific issue, I was able to talk to the speaker and one of his colleagues, who were as baffled as me about this issue at first, but then gave me a rather convincing explanation.

The issue has multiple parts.

The first problem is overdraw. The material is flagged as opaque, and opaque materials shouldn’t be able to cause overdraw (as polygons behind them are discarded in the so called “z-test”). However, the alpha cutoff flag changes this.

Alpha cutoff makes it so that the texture becomes completely invisible in places, where the transparency of a texture (the alpha value) goes below a specific value (the cutoff value). This is great for plant textures, as then the model only needs a flat plane (-> low poly), and the texture makes it seem like the model has way more detail than it actually is modelled with.

This messes with the z-test however. Whenever a polygon with transparency is detected, the polygons behind doesn’t get discarded and gets drawn anyway. Hence, overdraw.

This leads me directly to the second issue, which seems paradox at first: The model is too low poly.

Funnily enough, if your model has too few vertices for an area on the screen, and transparency (or alpha cutoff) is present, this area will be completely redrawn, causing a lot of overdraw. The tree model I am using happens to have relatively big polygons for its leaves, with multiple layers of leaves stacked on top of each other. Combined with me scaling the tree models into oblivion, big parts of the screen are redrawn 5 times. On pc not noticeable (some optimization magic?), but mobile gets hit pretty strongly.

And this is connected to the last issue: Most mobile GPUs are using a technique called “Tile based rendering”. I haven’t wrapped my head around this tech yet, but essentially it splits the screen into multiple areas (-> tiles), which are individually processed by the GPU.

From what I understand, if there is few geometry spanning multiple tiles, this technique performs the best (with big vertices those many tiles will have to be redrawn for each overdraw instance). Also, the tiles seem to be optimized for vertical phone orientation.

And my game misses both checks, being horizontal oriented and having quite some vertices that individually take a lot of space on the screen, spanning many tiles.

So, what is the solution? My artist and I are currently experimenting in 2 ways:

Either, we scrap the alpha cutoff all together and create a more high poly tree. This might fix the overdraw, but could result in too much load for low end phones to handle.

Or we optimize the tree model, such that most of its leaves’ area is filled with completely opaque vertices, and only the edges are left with cutoff. This will still cause overdraw to happen, but hopefully way less than before.

I will update this once I got to test both solutions.