Homebrew [Release] Caelina - OpenGL Driver

[machinamentum]

There's a bit of a chicken-and-egg problem here. I'm sure some people would eventually be willing to contribute to this project even without having done any prior 3DS-related work before. There's no way people can prove their willingness towards contributing when everybody keeps the basis for themselves. Also note that lots of people are more interested in working on base libraries like this without actually developing applications themselves, so waiting for people to do something with your library might not be the best approach.

Providing source code gives people the ability to jump in and help you out (if not by developing new features, then maybe by spotting and fixing bugs which you otherwise might not have noticed), and there's no reason not to do it. And even if people don't end up helping you, looking at the source code might give them a few clues on how to develop other interesting stuff. For instance, imagine that the ctrulib or nihstro examples had only shipped as precompiled binaries without source code - judging by how similar the shader in the Caelina library is, wouldn't that have made your job quite a bit harder?



You have to understand that many developers stay away from closed-source library on principle, for good reasons. Having a reference implementation has nothing to do with that. Fear of malicious code (which lots of closed-source libraries are contaminated with one way or the other) is just one of the reasons, another one would be lack of customizability and lack of insight into implementation details (which is important when documentation is vague or the implementation turns out to be buggy).

Well, you have a lot of solid points. There's still a handful of changes I need to make and formatting fixes (Atom is an awful editor because of a weird tab bug). I'll put out the source sometime within the next week.

 

[neobrain]

Sounds good, I'm looking forward to it

 

[machinamentum]

I've pinned down another allocation bug in immediate mode drawing (not the display lists). It seems by using linearAlloc/linearFree over and over, eventually, linearAlloc stops giving unallocated memory regions (doesn't seem to sanely manage the memory pool for our purposes), thus soft-locking the GPU. So, alternatively, I think it'd be best to allocate a substantial portion of the heap that is then managed within the driver itself on a per-context basis. Based on my current practices with the GL, we'd need 2 contexts in order to render to both screens. The question being, how much memory is appropriate for the driver to claim before getting greedy with resources?

EDIT: after extensive testing, allocation doesn't seem to be the issue anymore.

 

[shinyquagsire23]

I've pinned down another allocation bug in immediate mode drawing (not the display lists). It seems by using linearAlloc/linearFree over and over, eventually, linearAlloc stops giving unallocated memory regions (doesn't seem to sanely manage the memory pool for our purposes), thus soft-locking the GPU. So, alternatively, I think it'd be best to allocate a substantial portion of the heap that is then managed within the driver itself on a per-context basis. Based on my current practices with the GL, we'd need 2 contexts in order to render to both screens. The question being, how much memory is appropriate for the driver to claim before getting greedy with resources?

I've encountered this crashing my 3DS after a while, it'd be good to fix this.

 

[machinamentum]

I've encountered this crashing my 3DS after a while, it'd be good to fix this.

It's high on my TODO list, so it'll be fixed fairly soon. For the time being, if you cache all your vertices into display lists then draw using glCallList, this shouldn't happen and your code will execute 100x* faster.

*Approximate guess, but virtually, it's that much faster

Also, the source is now publicly available at https://github.com/machinamentum/Caelina

 

[KingOfHell]

I've pinned down another allocation bug in immediate mode drawing (not the display lists). It seems by using linearAlloc/linearFree over and over, eventually, linearAlloc stops giving unallocated memory regions (doesn't seem to sanely manage the memory pool for our purposes), thus soft-locking the GPU. So, alternatively, I think it'd be best to allocate a substantial portion of the heap that is then managed within the driver itself on a per-context basis. Based on my current practices with the GL, we'd need 2 contexts in order to render to both screens. The question being, how much memory is appropriate for the driver to claim before getting greedy with resources?

I've pinned down another allocation bug in immediate mode drawing (not the display lists). It seems by using linearAlloc/linearFree over and over, eventually, linearAlloc stops giving unallocated memory regions (doesn't seem to sanely manage the memory pool for our purposes), thus soft-locking the GPU. So, alternatively, I think it'd be best to allocate a substantial portion of the heap that is then managed within the driver itself on a per-context basis. Based on my current practices with the GL, we'd need 2 contexts in order to render to both screens. The question being, how much memory is appropriate for the driver to claim before getting greedy with resources?

Haven't closely looked that closely, but wouldn't that be a bug in ctrulib? Also, if it's on a per-context basis, wouldn't that mean you'd have to allocate something twice if you want it drawn on both screens?

 

[machinamentum]

Haven't closely looked that closely, but wouldn't that be a bug in ctrulib?

I don't know. I have to put in a bit of faith that ctrulib is working correctly and that there's a bug in my code.

Also, if it's on a per-context basis, wouldn't that mean you'd have to allocate something twice if you want it drawn on both screens?

Yes and no. The specification only defines how a display list is supposed to work, but not how it is bound. Most desktop implementations allow the user to enable or disable display list sharing among contexts. This seems like it'd be simple to add to the library so i'll probably implement it. You'd only have to setup your state (matrices, lights, etc...) separately.

Drawing without a display list would allocate multiple times.

 

[Idaho]

I hope this will help devs to bring Quake games over the 3DS

 

[machinamentum]

I've encountered this crashing my 3DS after a while, it'd be good to fix this.

Interestingly enough, switching sbuffer's allocator from linearAlloc/linearFree to new/delete again seems to have remedied the issue...

At this point, I'll just accept that it seems to work by calling linearAlloc as little as possible and using malloc for general data.

EDIT: still freezes at some point, investigating methods of at least not getting stuck in a soft-lock.
EDIT2: according to linearSpaceFree(), no heap space is being re-allocated now, so I can't say for sure what's causing the crash.

 

[Selim873]

I hope this will help devs to bring Quake games over the 3DS

Dude, if I had the skills, I would have totally tried this! Quake was actually the first thing I thought of when I saw this thread! I would LOVE to play through Quake 1 on my N3DS! I played it on my good ol' blue Phat NDS on my R4, and that was WITHOUT the EZ 3-in-1 RAM Expansion Pack, so I don't see how it would be impossible!

 

[elhobbs]

i think there is an issue with ctrulib 3d in general at this point. Most non-trivial 3d homebrew replace gspWaitForEvent with a version with a timeout in order to avoid these locks. This is what I ended up doing - a bit of a hack but it works. https://github.com//spectre3ds/blob/master/arm11/source/sys_3ds.cpp

Also selim873 - if you are adventurous you might try the "save" branch from the link above. It is a little tricky to setup and there is no menu - console only. But it is playable. If you can't figure it out let me know.

 

[hippy dave]

https://github.com//spectre3ds/blob/master/arm11/source/sys_3ds.cpp

404...

 

[zoogie]

404...

Somehow he left the elhobbs out bashful i guess
https://github.com/elhobbs/spectre3ds/blob/master/arm11/source/sys_3ds.cpp

 

[hippy dave]

Somehow he left the elhobbs out bashful i guess
https://github.com/elhobbs/spectre3ds/blob/master/arm11/source/sys_3ds.cpp

Ooh quake, does it run yet?
e: nm, just re-read his post above. Interesting.

 

[machinamentum]

i think there is an issue with ctrulib 3d in general at this point. Most non-trivial 3d homebrew replace gspWaitForEvent with a version with a timeout in order to avoid these locks. This is what I ended up doing - a bit of a hack but it works. https://github.com//spectre3ds/blob/master/arm11/source/sys_3ds.cpp

I'm definitely considering this as an option, but I feel like it's overall hacky and ignores the fact that something is fundamentally wrong with how we're using the GPU if by pseudo-random chance it's hanging during normal use. Seems even smea's gs code does this, so I might as well do the same.

 

[elhobbs]

I'm definitely considering this as an option, but I feel like it's overall hacky and ignores the fact that something is fundamentally wrong with how we're using the GPU if by pseudo-random chance it's hanging during normal use. Seems even smea's gs code does this, so I might as well do the same.

It is definitely a hack - but a nesacary one at this point. I think blargsnes is the original source. Forget who I stole it from at this point. The actual situation happens infrequently so it is not that bad.

 

[machinamentum]

It is definitely a hack - but a nesacary one at this point. I think

At this point, if it fixes non-cached immediate mode in the library, I'll take it. It's still odd though that this issue hasn't affected rendering buffers that have static data (in my testing at least, I've had a demo run for hours without any issue when using only display lists while otherwise a demo would run only minutes before locking).

 

[elhobbs]

Ooh quake, does it run yet?
e: nm, just re-read his post above. Interesting.

Somehow dropped while pasting the link from my phone. Yes, it runs. Needs pak files in an id1 directory. This is relative to the 3dsx file for ninjhax or in the root for cia or 3ds.

 

[Selim873]

i think there is an issue with ctrulib 3d in general at this point. Most non-trivial 3d homebrew replace gspWaitForEvent with a version with a timeout in order to avoid these locks. This is what I ended up doing - a bit of a hack but it works. https://github.com//spectre3ds/blob/master/arm11/source/sys_3ds.cpp

Also selim873 - if you are adventurous you might try the "save" branch from the link above. It is a little tricky to setup and there is no menu - console only. But it is playable. If you can't figure it out let me know.

I would love to, but I won't be getting my copy of Cubic Ninja for another week or so. Hahaha Definitely something I'll look into on a weekend when I have time though!! I'll PM you if I get stuck.

 

[machinamentum]

I've encountered this crashing my 3DS after a while, it'd be good to fix this.

It should (hopefully) be fixed now. Pull the latest commits and build from source.