Unity 3DS shader

Discussion in '3DS - Homebrew Development and Emulators' started by dotaku, Apr 14, 2018.

  1. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    Hi :)

    I am currently working on making my own 3DS game in unity. I got everything working but there is one thing I just can't figure out by googling.
    I wanna use cell shading for my game and I just cant get it to work. Vertex lighting works like a charm but every shader I make that is not just Vertexlit works in unity but does not work when run on my 3DS.

    Can anyone help me with how to make such a shader for the 3DS or give me some information on what to consider to make a shader work on the 3DS. Also I am using the old 3DS, Thanks for the help would be much appreciated! :)
     
  2. niin401

    niin401 Newbie

    Newcomer
    2
    Dec 26, 2009
    United States
    Try asking on the Nintendo Developer Portal forums, you'd probably get better Unity for 3DS support there. I am also working on a 3DS game with Unity but I am not using cell shading at all. I can't remember exactly what it is since I haven't worked on the game in a while, but there is a way to tell if your shader is going to run on the 3DS without actually running it on the 3DS or through an emulator.
     
    dotaku likes this.
  3. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    Thanks for your reply! I was looking exactly for such forum but did not find a "Nintendo Developer Portal forum" that's why I ended up posting here. I just looked again and I can't find it. Does it really exist or am I blind? :D
     
  4. Voxel

    Voxel Man, I'm beat!

    Member
    13
    GBAtemp Patron
    Voxel is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    Jun 27, 2015
    United Kingdom
    Many cel-shading techniques rely heavily on the use of fragment shaders. Unfortunately, the 3DS's GPU has extremely limited fragment shader support, and instead uses old texture combiners, so it really makes advanced shader programming on the 3DS a real pain in the ass!

    Luckily, there's a good way I've devised myself, that I used to use a lot in the past, to achieve cel-shading. I don't really use Unity for 3DS much anymore, so I'll explain it in some detail so that you and others can use it as you please.
    Basically, it's just a 2-pass shader which first draws a color or a texture, then applies a 1-dimensional toon ramp texture based on lighting and vertex normal positions. The toon ramp can have as many tones as you want it to have, but I prefer to start off with 2 (if you need an example toon ramp to use, you can use mine here) it must ideally be grayscale and transparent to get as close to actual cel-shading as possible.

    The first pass looks a little something like this; it's really not far off from looking like a regular unlit shader so this should be easy enough:
    Code:
    // Unlit Texture Pass
           Pass{
               Tags{ "LightMode" = "ForwardBase" }
               Blend SrcAlpha OneMinusSrcAlpha
             
               CGPROGRAM
    #pragma vertex vert
    #pragma fragment frag
    #include "UnityCG.cginc"
    
               struct appdata {
                   float4 vertex : POSITION;
                   float2 texcoord : TEXCOORD0;
               };
    
               struct v2f {
                   float2 texcoord : TEXCOORD0;
                   float4 vertex : SV_POSITION;
               };
    
               sampler2D _MainTex;
               float4 _MainTex_ST;
               float4 _Color;
    
               v2f vert(appdata v) {
                   v2f o;
                   o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                   o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
                   return o;
               }
    
               float4 frag(v2f i) : COLOR {
                   float4 col = _Color * tex2D(_MainTex, i.texcoord);
                   return col;
               }
               ENDCG
           }
    
    Then you have your second pass which applies the toon ramp based on lighting/normal positions, which looks like this:

    Code:
    // Toon Ramp pass
            Pass {
               Tags{ "LightMode" = "ForwardBase" }
               Blend SrcAlpha OneMinusSrcAlpha
    
               CGPROGRAM
    #pragma vertex vert
    #pragma fragment frag
    #include "UnityCG.cginc"
    
               struct appdata {
                   float4 vertex : POSITION;
                   float3 normalVec : NORMAL;
                   float2 texcoord : TEXCOORD0;
               };
    
               struct v2f {
                   float2 texcoord : TEXCOORD0;
                   float4 vertex : SV_POSITION;
               };
    
               sampler2D _RampTex;
               float4 _RampTex_ST;
    
               v2f vert(appdata v) {
                   v2f o;
                   o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                   o.texcoord = TRANSFORM_TEX(v.texcoord, _RampTex) + dot(mul(float4(v.normalVec, 0.0), _World2Object).xyz, _WorldSpaceLightPos0);
                   return o;
               }
    
               float4 frag(v2f i) : COLOR {
                   float4 col = tex2D(_RampTex, i.texcoord);
                   return col;
               }
               ENDCG
            }
    
    It's worth noting for future reference; when you're trying to create your own shaders for 3DS, you can click on your shader in the Unity file window, then click on "Compile and show code" from the inspector, and if it contains some compiled code, then it may work. Otherwise, if you get something like an error, or completely empty code, it's unsupported.

    Hope this clears some things up for you! Just a little extra thing, outline shaders are very buggy on 3DS right now, so it's likely you'll have to go without any sort of outlining unless you're planning on making a game that doesn't make use of stereoscopic 3D.
     
    emuashui, KiiWii, x65943 and 2 others like this.
  5. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    Thank you so much! :D

    But for some reason I cant get the second pass to work first one looks like a unlit shader(as intended). But second pass Is not reacting to light and is just grey when tested alone. I am using the ramp texture you send so this should not be the problem. Here the full shader code, maybe I did something wrong I am very new at making shader :D :

    Code:
    Shader "Custom/Cellshader"
    {
        Properties {
            _Color("Main Color", Color) = (0.5,0.5,0.5,1)
            _MainTex("Base (RGB)", 2D) = "white" {}
            _Ramp("Ramp (RGB)", 2D) = "gray" {}
        }
        SubShader{
            Tags { "RenderType" = "Opaque" }
            LOD 80
            // Non-lightmapped
            // Unlit Texture Pass
            Pass{
                Tags{ "LightMode" = "ForwardBase" }
                Blend SrcAlpha OneMinusSrcAlpha
                CGPROGRAM
                #pragma vertex vert
                #pragma fragment frag
                #include "UnityCG.cginc"
                struct appdata {
                float4 vertex : POSITION;
                float2 texcoord : TEXCOORD0;
                };
                struct v2f {
                float2 texcoord : TEXCOORD0;
                float4 vertex : SV_POSITION;
                };
                sampler2D _MainTex;
                float4 _MainTex_ST;
                float4 _Color;
     
                v2f vert(appdata v) {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
                return o;
                }
                float4 frag(v2f i) : COLOR{
                float4 col = _Color * tex2D(_MainTex, i.texcoord);
                return col;
                }
                ENDCG
            }
            Pass{
                Tags{ "LightMode" = "ForwardBase" }
                Blend SrcAlpha OneMinusSrcAlpha
                CGPROGRAM
                #pragma vertex vert
                #pragma fragment frag
                #include "UnityCG.cginc"
                struct appdata {
                float4 vertex : POSITION;
                float3 normalVec : NORMAL;
                float2 texcoord : TEXCOORD0;
                };
                struct v2f {
                float2 texcoord : TEXCOORD0;
                float4 vertex : SV_POSITION;
                };
                sampler2D _RampTex;
                float4 _RampTex_ST;
                v2f vert(appdata v) {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                o.texcoord = TRANSFORM_TEX(v.texcoord, _RampTex) + dot(mul(float4(v.normalVec, 0.0), unity_WorldToObject).xyz, _WorldSpaceLightPos0);
                return o;
                }
                float4 frag(v2f i) : COLOR{
                float4 col = tex2D(_RampTex, i.texcoord);
                return col;
                }
                ENDCG
            }
        }
    }
    
    
     
    Last edited by dotaku, Apr 14, 2018
  6. Voxel

    Voxel Man, I'm beat!

    Member
    13
    GBAtemp Patron
    Voxel is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    Jun 27, 2015
    United Kingdom
    In your properties, "_Ramp" should be "_RampTex", otherwise all the "_RampTex" code in the 2nd pass goes unused.
    Also, I forgot to mention this, but you'll also need to tweak the toon ramp texture settings as follows:

    upload_2018-4-14_20-3-39.

    After that, everything should all work nicely! :)
     
    dotaku likes this.
  7. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    oh what a silly mistake of mine sorry!

    Its working now! But not fully as expected. On a sphere it is working for example. But not on the model I use for testing shades. Is this just a limitation of this shader to not work on more complex shapes or is something still going wrong?
    I saw the shader is depending on UV and normals which both should be correct in this model

    [​IMG]

    The one on the top is what I was aiming for
     
    Last edited by dotaku, Apr 14, 2018
    Voxel likes this.
  8. Voxel

    Voxel Man, I'm beat!

    Member
    13
    GBAtemp Patron
    Voxel is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    Jun 27, 2015
    United Kingdom
    Yeah, unfortunately the results can vary with this shader, and I've had to tweak my models sometimes to adapt to it, but this is pretty much the only existing workaround that I know of. :(
     
    dotaku likes this.
  9. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    Well anyway Thanks a lot! This brought me much further than I would ever have gotten on my own! :)

    Also maybe something is wrong with my model after all. I tested on a other rather complex model and there the result is quite decent.
    [​IMG] [​IMG]
    [​IMG]

    Hmm I wounder whats wrong with my first model that makes it behave so wrong. The first model is not even changing at all when I rotate the light.

    Edit: Seems like my model had a scale off 100 and the shader did not like that. I changed that and now I am happy with how it looks! I am sure I can get it to look exactly how I want by tweaking the ramp :D

    Thanks you so much again Voxel you helped me out a lot!! :)
     
    Last edited by dotaku, Apr 14, 2018
    Voxel likes this.
  10. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    oh damn! No idea why but the shader is not working when on the 3ds :(
    Now I really am confused. Are there other things to consider to make it work on the 3DS. Could it be that it does not work because I use a Old 3DS?

    Here the shader code after compilation maybe that helps because the "// shader disassembly not supported on n3ds" worrys me. But I use exectly the code posted some posts ago:
    Code:
    // Compiled shader for custom platforms, uncompressed size: 1.1KB
    Shader "Custom/Cellshader" {
    Properties {
     _Color ("Main Color", Color) = (0.500000,0.500000,0.500000,1.000000)
     _MainTex ("Base (RGB)", 2D) = "white" { }
     _RampTex ("Ramp (RGB)", 2D) = "gray" { }
    }
    SubShader {
     LOD 80
     Tags { "RenderType"="Opaque" }
     Pass {
      Tags { "LIGHTMODE"="ForwardBase" "RenderType"="Opaque" }
      Blend SrcAlpha OneMinusSrcAlpha
      GpuProgramID 5745
    Program "vp" {
    SubProgram "n3ds " {
    Bind "vertex" Vertex
    Bind "texcoord" TexCoord0
    Matrix 0 [glstate_matrix_mvp]
    Vector 4 [_MainTex_ST]
    SetTexture 0 [_MainTex] 2D 0
    "// shader disassembly not supported on n3ds"
    }
    }
    Program "fp" {
    SubProgram "n3ds " {
    ""
    }
    }
     }
     Pass {
      Tags { "LIGHTMODE"="ForwardBase" "RenderType"="Opaque" }
      Blend SrcAlpha OneMinusSrcAlpha
      GpuProgramID 101699
    Program "vp" {
    SubProgram "n3ds " {
    Bind "vertex" Vertex
    Bind "normal" Normal
    Bind "texcoord" TexCoord0
    Matrix 0 [glstate_matrix_mvp]
    Matrix 4 [unity_WorldToObject] 3
    Vector 8 [_RampTex_ST]
    Vector 7 [_WorldSpaceLightPos0]
    SetTexture 0 [_RampTex] 2D 0
    "// shader disassembly not supported on n3ds"
    }
    }
    Program "fp" {
    SubProgram "n3ds " {
    ""
    }
    }
     }
    }
    }
    
     
    Last edited by dotaku, Apr 15, 2018
  11. HaloBenish

    HaloBenish GBAtemp Fan

    Member
    3
    Feb 1, 2008
    Canada
    Login to developer.nintendo.com and in the 3DS section there is a forum underneath downloads I believe.
     
    dotaku likes this.
  12. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    I fixed it in case any one else sees this and has the issue you have to change "Tags{ "LightMode" = "ForwardBase" }" in both passes to Tags{ "LightMode" = "Vertex" }
     
  13. Voxel

    Voxel Man, I'm beat!

    Member
    13
    GBAtemp Patron
    Voxel is a Patron of GBAtemp and is helping us stay independent!

    Our Patreon
    Jun 27, 2015
    United Kingdom
    Lol well I'm glad that you managed to fix it; I tried to look into it as well this morning but the shader never worked for me either... which is really, really weird because it works in another 3DS project I've got currently made. :unsure:
    Unity 5.4 3DS seems to have hit-and-miss results with custom-made shaders sometimes, I know a couple of others who had difficulty with it in the past. When I was trying to fix the shader this morning, re-importing everything and recompiling caused pretty much every single shader (including default ones) to completely break on real hardware! :(
     
    dotaku likes this.
  14. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    Yea it really is a little pain in the a** :D But I am happy I was able to get what I wanted thanks to you :)

    I even managed to make it look exactly like I wanted by changing your shader. Here how it turned out:
    [​IMG]

    And yeay it runs on my 3ds :D

    Here the code for the second pass first one stays the same:


    Warning: Spoilers inside!
     
    Last edited by dotaku, Apr 15, 2018
    Voxel likes this.
  15. xXDungeon_CrawlerXx

    xXDungeon_CrawlerXx GBAtemp Advanced Maniac

    Member
    9
    Jul 29, 2015
    Liverpool
    Would you mind to share the full shader with us?
     
    dotaku likes this.
  16. dotaku
    OP

    dotaku Member

    Newcomer
    2
    Apr 14, 2018
    Germany
    Yes, you are right that would probably make it more easy for some people. It's not clean yet but here you go:


    Code:
    Shader "Toon/Basic"
    {
        Properties
        {
            _MainTex ("Detail", 2D) = "white" {}                                       
            _ToonShade ("Shade", 2D) = "white" {}                                         
            _Color ("Base Color", Color) = (1,1,1,1)                                             
            _Brightness ("Brightness 1 = neutral", range(0,2)) = 1.0                  
    
            _X("rot light X", range(0,3.141)) = 0
            _Y("rot light Y", range(0,3.141)) = 0
            _Z("rot light Z", range(0,6.282)) = 0
        }
     
        Subshader
        {
            Tags { "RenderType"="Opaque" }
            Blend One One
            LOD 250
            ZWrite On
               Cull Back
            Lighting Off
            Fog { Mode Off }
    
                Pass{
                Tags{ "LightMode" = "Vertex" }
                Blend SrcAlpha OneMinusSrcAlpha
    
                CGPROGRAM
    #pragma vertex vert
    #pragma fragment frag
    #include "UnityCG.cginc"
    
                struct appdata {
                float4 vertex : POSITION;
                float2 texcoord : TEXCOORD0;
            };
    
            struct v2f {
                float2 texcoord : TEXCOORD0;
                float4 vertex : SV_POSITION;
            };
    
            sampler2D _MainTex;
            float4 _MainTex_ST;
            float4 _Color;
    
            v2f vert(appdata v) {
                v2f o;
                o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
                o.texcoord = TRANSFORM_TEX(v.texcoord, _MainTex);
                return o;
            }
    
            float4 frag(v2f i) : COLOR{
                float4 col = _Color * tex2D(_MainTex, i.texcoord);
                return col;
            }
                ENDCG
            }
           
            Pass
            {    
                Tags{ "LightMode" = "Vertex" }
                Blend DstColor SrcColor
                CGPROGRAM
                    #pragma vertex vert
                    #pragma fragment frag
    
                    #include "UnityCG.cginc"          
    
                    sampler2D _MainTex;
                    half4 _MainTex_ST;
                    float _X, _Y, _Z;
                    sampler2D _ToonShade;
                    float _Brightness;
                   
                    struct appdata
                    {
                        float4 vertex : POSITION;
                        float3 normal : NORMAL;
                        float4 texcoord : TEXCOORD0;
                    };
                   
                     struct v2f
                     {
                        float4 pos : SV_POSITION;
    
                        float2 texcoord : TEXCOORD0;
                     };
                 
                    v2f vert (appdata v)
                    {
                        v2f o;
    
                        o.pos = mul ( UNITY_MATRIX_MVP, v.vertex );
    
                        float3x3 rotMatX = float3x3(1, 0, 0, 0, cos(_X), -sin(_X), 0, sin(_X), cos(_X));
    
                        float3x3 rotMatY = float3x3(cos(_Y), 0, sin(_Y), 0, 1, 0, -sin(_Y), 0, cos(_Y));
    
                        float3x3 rotMatZ = float3x3(cos(_Z), -sin(_Z), 0, sin(_Z), cos(_Z), 0, 0, 0, 1);
    
    
                        v.normal = mul(v.normal, rotMatX);
                        v.normal = mul(v.normal, rotMatY);
                        v.normal = mul(v.normal, rotMatZ);
    
                   
    
    
                        float3 n = mul((float3x3)UNITY_MATRIX_IT_MV, v.normal);
                        normalize(n);
                        n = n * float3(0.5,0.5,0.5) + float3(0.5,0.5,0.5);
                        o.texcoord = n.xy;
    
                        return o;
                    }
    
                     
         
                   
                   
                    fixed4 frag (v2f i) : COLOR
                    {
                        fixed4 col = tex2D( _ToonShade, i.texcoord);
                        return  col* _Brightness;
                    }
                ENDCG
            }
        }
        Fallback off
    }
     
  17. xXDungeon_CrawlerXx

    xXDungeon_CrawlerXx GBAtemp Advanced Maniac

    Member
    9
    Jul 29, 2015
    Liverpool
    Thanks a lot. Looks good!
     
    dotaku likes this.
  18. HaloBenish

    HaloBenish GBAtemp Fan

    Member
    3
    Feb 1, 2008
    Canada
Loading...