Newer libfat slower than 5-year old version?

Discussion in 'NDS - Emulation and Homebrew' started by DSLSC, Nov 8, 2013.

  1. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    The source code for Jeremy Ruten's dsBible is freely available at http://viewsourcecode.org/homebrew/ds-bible. Upon finding it, I decided to add some changes and improvements. It was enough of a challenge to update the program to compile with the latest devKitPro build and libfat access commands. However, although the program still functions, the libfat operations go SO SLOW! The original 2008 compilation loads in 14 seconds (I was hoping to reduce that), and is 228KB in size...but my 2013 compilation is 344KB, and takes 1 minute and 44 seconds to load. I know that "newer=better", and usually "newer=bigger", but does "newer=slower"? Or is something wrong?

    I should note that I am using a Supercard Mini SD, and also that the devKitPro "libfat" example does not work on the DS, even when DLDI-patched. (When unpatched, it says, "fatInitDefault failure: terminating." When patched, it says, "opendir() failure; terminating.")
    The "2013 dsBible" build functions only if it is patched with the scsd_moon (Moonshell) DLDI patch(!) Any clues?

    P.S. I've successfully made a dsBible port of the NASB translation. If anyone's interested, I'll post it.
     
  2. Coto

    Coto GBAtemp Addict

    Member
    2,367
    416
    Jun 4, 2010
    Chile
    maybe the SD interface driver is in a tight loop or something because overlapping values (interface offsets) in newer libfat distros..
     
  3. CalebW

    CalebW Fellow Temper

    Member
    634
    154
    Jun 29, 2012
    United States
    Hey, while you're at it...could you make it where DSBible shows more than the first 300 results when you do a search, and make it where after you click on a search result and then go back to the search list it memorizes where you are instead of taking you back to the beginning? Cause it's a pain in the but to have to scroll through 50+ searches to get to the one you want every time...
     
  4. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Coto: I'm not sure that I understand exactly what you're referring to. I'm assuming that it is something inside of libfat, which is out of my reach. The loop that takes 104 seconds (new version) to load a translation is simply searching through the entire file to find the index offset of each book. Obviously, if there was an index location header in the translation file, the entire loop would immediately become obsolete...
    What really makes me suspicious is that the official devKitPro libfat example doesn't work on the DS, even when DLDI patched. I would note that the "2013 dsBible" runs slowly on DeSmuME as well. Should I look for an older version of libfat? (And/or devKitPro?) Or is there some recent bug in libfat that I should wait for a fix?

    CalebW:Actually, most of my intended improvements have to do with the search feature. A quick, partial list of improvements is:
    • Use the devKitPro keyboard for Search and Verse Lookup
    • Support searching for multiple search terms with AND capability (currently, it's just OR)
    • Display a small snippet of the text around each result in the results list. Sure, and remember where you were.
    • Make Verse Lookup a little easier to use
    • Speed it up (!)
    However, Rule No.1 for improving something is: Don't make it worse :rolleyes:. Which is the difficulty I am having now with the 1.75-minute startup time. Suddenly, 14 seconds doesn't seem so bad...
     
  5. Coto

    Coto GBAtemp Addict

    Member
    2,367
    416
    Jun 4, 2010
    Chile
    Then a cache of some sort would be useful.

    ie: every book once open the first time creates an UID. then it goes to a small cache field.

    everytime the app starts, it will look from the last index ID from book UID and will look for further index IDs. If none is found then it's ready to go.
    if cache reaches a certain length, cache will split-up in de-referenced UIDs (UID:1->11-1f)

    Think of this as a measure to prevent giant books from causing hangups, and / or libfat changes.


    (it will still require DLDI, as direct flashcart calls and ram entrypoints are in macros inside libfat).
     
  6. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    A cache would be useful, sure, but the data still has to be loaded. So far, the only file command that's is faster with the newer libfat (as far as I can tell) is "fatInitDefault()". As I had to change some libfat access commands to make the program compile in the current version of devKitPro, I am wondering if that has something to do with it. (For example: "diropen" -> "opendir", and changed datatypes.)

    For a complete Bible search, though, the entire file needs searched...and taking nearly 2 minutes for that is excessive. Is anyone using the current devKitPro libfat in a method that requires a high data rate? Is "fread" faster than "fgets"? Or is this the wrong place to ask such questions?

    P.S. Coto: I don't quite understand what you're getting at with UIDs. I certainly do intend to make an optional "byte offset index" table at the head of each translation file to speed up the load process (if that table was present in the file, dsBible wouldn't need 14 seconds to create the table at startup), but need to figure out what's going on with libfat FIRST.
     
  7. Normmatt

    Normmatt Former AKAIO Programmer

    Member
    2,142
    544
    Dec 14, 2004
    New Zealand
    fread is always faster than fgets....
     
  8. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Maybe after I can get it working, that is ;). fgets is being used to read 1024 bytes at a time from the file into a char array. I do note that dsBible reads 277KB of bitmaps into memory with fread, and that takes less than a second. It takes almost 2 minutes to read through 4MB of plain text with fgets. I tried fread, and it only read a few chunks before locking up. 4MB/277KB = 14...Wait a minute...that's how many seconds it should take! Thanks Normmatt. Will try to get fread working, and will keep wondering what happened to fgets in 5 years of libfat improvements...

    Would NitroFS be faster than libfat (now is)? I haven't tried the devKitPro NitroFS example yet...wonder if it will work on the NDS on a Supercard Mini SD.
     
  9. Normmatt

    Normmatt Former AKAIO Programmer

    Member
    2,142
    544
    Dec 14, 2004
    New Zealand
    I'm not sure what happened to fgets to make it slower.. but its not libfat related its more than library a bug in the standard library which provides fgets.
     
  10. Foxi4

    Foxi4 On the hunt...

    pip Reporter
    23,558
    21,536
    Sep 13, 2009
    Poland
    Gaming Grotto
    14 seconds for startup sounds absolutely ridiculous, are you absolutely sure that it's libfat that's at fault? That, and why even use FAT or NitroFS when the entirety of the bible is lower than 4MB (around a half of that, I just checked)? You can literally include the whole thing without having to use either.
     
  11. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Normmatt: It appears that in the current version of libfat, each time "fgets" is called, it (re)loads the sectors from the FAT device, instead of checking to see if the requested read has already been cached. That's my best guess. So far, my attempts to make an "fgets" out of "fread" haven't been successful. As I have found, fgets is for plain text input, where it gets one line from the file on each call. fread just gets "x" number of bytes. Are there any special-input functions, or should I take this up on the devKitPro forums?

    Foxi4: A complete uncompressed Bible translation is just over 4MB. If I wrote my own Bible program, it would most likely compress the files with PuCrunch, which would reduce the file size by over 50%. It would also have a "book index" header, which would remove the startup delay. In Jeremy Ruten's last published version, the libfat initialization takes about a second, and then it slowly reads the entire Bible file to locate the books. But updating Ruten's dsBible is completely different than making my own. Which I may end up doing...or not :rolleyes:.
    Yes, the translations can easily be included into the NDS binary. But then you can't add translations, edit, correct, etc. At least not without a recompilation. I have considered including the GUI bitmaps into the binary, though.
     
  12. Foxi4

    Foxi4 On the hunt...

    pip Reporter
    23,558
    21,536
    Sep 13, 2009
    Poland
    Gaming Grotto
    Consider splitting the .txt file into individual books, replacing FAT reads with NitroFS reads, embedding the files into NitroFS and only loading the index pages upon initing - that way, you should be golden boot-wise.
     
  13. DSLSC
    OP

    DSLSC Member

    Newcomer
    28
    3
    Aug 27, 2013
    United States
    Embedding the files into NitroFS? As I gather from the devKitPro example, NitroFS is a lot like GBFS for the GBA. However, as I noted before, there's a difference between writing my own dsBible, and updating Jeremy Ruten's. I'm not sure which track to take—I like the dsBible logo :unsure:, but there's not too much that I can modify without it being a totally different program. However, I definitely would like to keep the translation files external from the main NDS file (making it easy to update, add, or modify translations), and preferably not "Filecount=Bibles*66". Yes that would do away with the requisite for an index, but...

    Maybe I should just leave the ~2-minute startup time for non-indexed files and work on an optional file index...or try to figure out how to make dsBible do a one-time-index process (without trashing the file). Maybe someone here knows how to insert some data into the beginning of a file without rewriting the entire thing?
    At any rate, thanks for all the suggestions.
     
  14. Coto

    Coto GBAtemp Addict

    Member
    2,367
    416
    Jun 4, 2010
    Chile
    (its been a while)

    Yes, the dldi site shows functions with different names for some reason. I didn't explicitly referred to the issue you had originally, I apologize.

    Well, I use dldi for some stuff but haven't had any issues...

    iirc fgets seeks the '\0' EOF whereas fread starts from base pointer until offset given run-size on blocksize chunks.

    fgets will be slower , but you may want to use an updated dldi for your card maybe? or maybe the cluster defaults for libfat does not match your currents cards clustersize.. who knows