Making of Varga by Paul Tosca
4.4.Further tweaking of normal maps in photoshop
This paragraph is for those of you who want to better understand normal maps and it may seem a little technical (it requires some basic knowledge of math , vectors and stuff like that ).
4.4.1.Some general info about TS (tangent space) normal maps
First, i would like to talk a little about normal maps , how they are computed/displayed.
I am more interested in tangent space normal maps ( world or object space normal maps are not suitable for characters or for objects that will deform ) so for now on when i will say normal map you will assume i am talking about tangent space unless stated otherwise.
If you are in the same situation i was when i first started with normal maps ( tangent space in particular ) probably the following info will help.
At the beginning i was really frustrated by the "Seam" problem :) : a normal map computed within a software will not display properly in another one , and i am talking here about the seam that will appear in the geometry where you will have an uv shell border.I have struggled to find out what the problem was ... thought i am doing something wrong , read the help of each apps then started searching on the internet and after some research i finally got the idea about the problem , how and why it will occur ...but not how to fix it though:)
The main problem resides in the way each application computes/displays tangent space normal maps.
Tangent space is represented by three vectors: normal ( that will always be perpendicular to the surface ) ,and two other vectors perpendicular to the normal called tangent and bi-normal ( or bi-tangent : i have seen that the names are used interchangeably though some say that the correct name will be bitangent since we are talking about a surface ... for a curve in 3D , the tangent space we will have one tangent and two normals , normal and binormal .... for surfaces we will have one normal and two tangent vectors : tangent and bitangent but the naming does not matter much as long as we know what vectors we are talking about). These three vectors can be defined for each point on a surface and taken all together they define a coordinate frame.
Usually tangent space per triangle will be computed in the following way:
-the normal vector will always be perpendicular to the surface (triangle) (it will be the blue one and it will be stored in the blue channel of the normal map)
-the tangent(the red one stored in red channel) and binormal(the green one stored in green channel) vectors will be oriented in accordance with the UVs :tangent will be left to right(or right to left) in UV space (so it will be U coordinate in texture space ) and binormal will be down to up (or up-down) (so it will be the V coord in texture space) ;probably you have seen in many dialogs for normal maps the options to flip Red or Green channel , these options stand for the orientations of the tangent(left-right/right-left) and binormal (up-down/down-up) .. some apps use different orientation than others; you can flip the channels inside photoshop also, for instance to flip the Red channel you will press Ctrl+1 ( to go to the red channel ) and Ctrl+I (to invert it) then Ctrl+~ (control tilda , that will be left to 1 key -) to go back to full RGB image (same for green Ctrl+2,Ctrl+I,Ctrl+~).
Now lets say that i will cut the UVs along the edge and will rotate one shell 90 degrees CCW like in the image below.Because of the rotation in UV space the tangent space for the first triangle will be different than in the first situation ( the RGB triad for the first triangle has rotated also 90 degrees along blue axis )
Because of this , the orientation of the uv shells will affect the colors you will see in the normal map;if you rotate the uvs of your model ( or the uvs for some shells ) lets say 90 degrees ClockWise and recompute the normal map the colors in the new normal map will be completely different ( not because the entire image will be rotated by 90 degrees ... you can rotate it 90 CCW in photoshop and compare it with the first version to check .... they will be different )
Tangent space per vertex will be then computed and after that you can compute the tangent space vectors for any arbitrary point on the surface by interpolating vertex tangent space vectors; usually more complex calculations are done so that the tangent space will be invariable regarding to tesselation (L shape problem) : for instance in previous versions of maya the computed normal map was dependent to tesselation ... so if you would have a triangulated model and if you would select one triangle edge and flip it then the normal map would no longer be displayed properly due to recalculation of TS after flipping the edge ... in the newer versions of maya this was solved ( usually special tools are used to prepare the geometry for per-pixel lighting by creating proper tangent basis at each vertex, NVMesh Mender is one of them)
Now ,for computing a normal map the raytracing algorithm will cast a ray from the current point of the low poly mesh to the hign poly and at the point of intersection with the high will compute the normal for the high surface in world space ( WS ) then this normal is transformed in the tangent space ( TS ) of the low poly version and stored in the normal map.So you see that the normal computed is dependent to the tangent space used for the low poly version and the tangent space normal map must be created using the same normal/tangent/binormal as the game uses ( or the application you will use to display the normal maps) ,otherwise the normals will be misinterpreted and you will see discontinuities on border uv shells , hence the problems you will have when trying to display a normal map computed with one app in another app.
It will not be a problem if different apps will use different methods for tangent space vector calculations , the problem is that they are not making that info available so that someone interested could write a plugin/tool that will remedy the problem ...the only app i know so far to made the formulae public is Mudbox , on their online help pages you can find how they are calculating tangent space vectors.
Now you know why the problem occur but there is little to do about it ... at least you should know that you aren't doing anything wrong , the app is to blame :) .
I will show an example and explain how the normal map will be computed and how to interpret it channel by channel... this will help if you want to alter the normal maps in photoshop , overlay multiple normal maps , paint them directly in photoshop etc ...
I will bake the normals from the plane and cylinder below to a single quad plane highlighted in green.As i explained earlier the tangent space will be : normal pointing up( blue one) tangent from left to right ( 3-4 red one) and bitangent/binormal down-up (3-1 the green one) . The cylinder has all the edges hard so the normals will change in steps, it will be easier to analyze.I have numbered the faces on the cylinder from 1 to 7.
