[Request Help] Latin characters in JP version of Diabolical Box

[MRJPGames]

Hi, someone reached out to me if I could help them with the technical side of translating Diabolical Box JP (as the JP version has a special "Holiday" mode missing from the international version).

Though only having experience with CV I did set out to help, 'cause rom hacking is fun, and is something I want to encourage.

However I can't really seem to solve the core problem: English text is not rendered.

So what did I try:
1.
Simply modifying the .txt files in the rom to contain English/Dutch text.
This does not crash the game. however this results in empty text boxes.

Spoiler: Empty text box

2.
Using Shift_JIS encoding for "Full Width" or JASCII letters.
This sorta works, problem is it's "Full width" which makes it highly impractical, and it looks pretty terrible.

Spoiler: Full width terribleness

3.
Attempting to replace Japanse characters that do render with latin letters in the font used (font.NFTR), then using shift_jis encoding for those Japanse characters.
This provides more or less the same result, unless I can find symbols that internally already have the same parameters set for width, distance between chars etc. (Which yes I did update in the font.NFTR file, but from what I can tell either Tinke does not update the file correctly (unlikely as the changes persist within Tinke upon full reload, or more likely the game simply disregards this part of the font data for whatever reason...)

Spoiler: A not so great font modification

Main difference between this and the Full Width version that can be done without font modification is that the letters are smaller here, but as Japanese characters tend to be rather wide comparatively they end up taking up (about?) the same amount of space.

I was able to find this old thread https://www.romhacking.net/forum/index.php?topic=30895.0 which might be wholy unrelated, however it too seemed to be a case of regular ASCII (just like first attempt mentioned in this post) not working. However I have not been able to find much on what "2 byte" encoding would be, other than that it might be referring to Full width, in which case due to seeming lack of effective font editing would really not be a great solution...

If any additional info is needed in order to help, let me know!

 

[FAST6191]

2 byte encoding is what you dubbed full width -- the shiftJIS table ( http://rikai.com/library/kanjitables/kanji_codes.sjis.shtml ) does feature Roman characters (and Greek/Cyrillic a bit later) in the 82?? range which means 16 bits/2 bytes per character. Officially ShiftJIS should support 8 bit encodings for said Roman letters but most games that feature it do not bother hence this, if it is a NFTR using title though then you might be able to add them.


NFTRedit might be a better bet for a tool to play with the font as well
https://gbatemp.net/download/nftredit.29196/
Think that was the latest version/fork that worked with a revision found in some very much later games but if not there is a thread around here somewhere (same name/title), and the change was minor enough you could change and back again with a hex editor.

Equally on "impractical" then other than maybe not being the best spaced (a real concern don't get me wrong) is there much impractical with it? Some games might store things in memory and face limitations that way if you start writing reams of text but that is going to be rare for a puzzle game. If it is forcing it then various Japanese IME/word processor type things will be able to force it trivially. JWPCE being the method of choice for many (free, easy enough to use) https://www.at-it-translator.com/jwpce/ but probably a bit long in the tooth right now, njstar was popular as well. I forget the specific usage but along the top row will be some buttons (three or four of them, one was an A, others some Japanese symboles, one is normal ASCII/Windows IME, others will do Japanese according to whatever scheme it uses to input and a final one should force said 16 bit encodings in addition to various Japanese offerings but you don't much care there.
Though some also opt to use https://www.romhacking.net/utilities/504/ in conjunction with the above if saving out is going to be annoying.
You might be able to convert things across from ASCII (or at least U16 unicode) by addition as well easily enough if you wanted to go that way.

 

[MRJPGames]

Thanks for the reply!

Those are indeed what I dubbed "Full Width" (afaik that's what unicode calls them?), and those are the latin (roman?) symbols that work with the default font.

As for font editing, I gave that tool a try and it's quite nice, but seems to do the same thing Tinke does. I've since the original post been able to confirm Tinke changes the font correctly (and/or at least the same way NFTRedit does). The problem is that Diabolical Box seems to disregard the metadata(?) entirely. As an easy example I changed a character's properties to all 0s, so 0 length, 0 width and 0 offset. The result:

(I had 4 of that character at the beginning for testing)
This makes me suspect that the game always uses the 14px length, that every used character has by default (except the special "Friendly Ban" version stuff on top, which is a separate font and likely not really useful here.)
The games does seem to use the width parameter though, as if I set it to a value lower than the width of the character only part of it is drawn (eg. set it to 3 and only the first three pixels are drawn, but the next character remains unmoved).
I haven't tested with just changing offset, as it shouldn't really help me afaik.

As for impracticality, it's really just about the spacing in-game. It wouldn't be too much trouble for me to write a little script that takes all the txt files in normal ASCII and converts it to this custom "full width" encoding.

Currently the way it's looking to me is that the likely somewhat easy solution (as I'm not skilled in binary editing) would be to find where the game stores this constant width, replace it with a smaller constant width that could be used for all roman/latin letters and than change the "Full Width" part of the font to look about right with that mono spacing and do it that way? Is there anything I might be missing, or does that also seem the most likely compromise between easiness of the work and functionality?
Or alternatively just take the hit, use the wide characters and deal with it. Biggest problem there would really just be the puzzle texts often taking up most of the top-screen when properly formatted, thus it would require shortening the content quite a bit... Which might make the puzzle harder to solve... Than again the aforementioned hack would have the same problem, just to a lesser extend.

 

[mn1712trungson]

This problem can be solved with Nintyfont and NFTR Edit.

First, you need to use Nintyfont to add the ASCII or Latin character map (if needed).

About the range of Unicode blocks.
Starting from any character is fine, but must continue until the end of the block.

In the glyph properties box, press ADD and enter the code point value (0x20 to 0x7E for ASCII).

Caution: don't import BITMAP value or font will broken.

Next, we check the font with Tinke to confirm the character map has loaded (type 0 for proportional, any character in type 2 will skip the offset value and become full-width characters; I not test type1). More information in: problemkaputt.de/gbatek-ds-cartridge-nitro-font-resource-format.htm

(ASCII character map loaded correctly - Type 0) (First char = decimal value in Unicode chart)

(Double check on offset 0Ch - 0x2 - value = 00 00 (type 0))

Now, you can draw tile with NFTR Edit or Crystaltile and settings width value.

Warning: please always backup your font. Anytime NFTR Edit can't open your modified font, it means your font is broken.

 

[MRJPGames]

I've been trying to do this for ~an hour. I'm running into some issues... However before I detail those, is it even worth continuing provided the following:
The font provided with the international (EU specifically) version of the game, which uses the same ASCII characters (though seemingly indeed better properly included in the font, which seems not to be the case in the JP version). However even when just straight up using that font instead of the default font, it just won't render anything. As that font has no issues in the EU/US versions (duh) I have my doubts if any modifications to the JP font will be successful.

As for problems I ran into trying your approach:
- The "Add" button in NintyFonts does absolutely nothing for me (I'm using "Second Alpha" release from GitHub)

However the font already includes all non-special latin letters (so é etc. not included, but alphabet, numbers and punctuation is).
So I tried adding a character map for them (not present by default!), I tried using Tinke for this, which did not work (Cast error).
In adition Tinke is not able to modify Character Map type from 1 to 0 (same/similar cast error). (I was hoping to simply update the type of FULL WIDTH chars, as they DO display, and hope to either modify those tiles, or point towards the pre-included ASCII char tiles.)


I'll be doing some more playing around with things on my end. Will update this comment with any additional details (if I find any).

Post automatically merged: Nov 18, 2022

So I did some more testing and it's making my rather confident that this is not an issue with the font but an issue with the way the game uses the font...
Here is a modified version of Diabolical Box JP and a modified version of Curious Village JP.
Modifications:
- Diabolical Box JP uses Curious Village JP's font.
- 1 Text box is changed in each game to:
"先生が、遺産相続騒動の解決を？
Roman Text Test!
Ｒｏｍａｎ　Ｔｅｘｔ　Ｔｅｓｔ！"

This is the result:

It should look exactly the same, but it does not. For whatever reason DB-JP doesn't want to render the regular ASCII characters, while CV-JP has no issue with them (though still being quite w i d e)

However I still have a bit of hope left for a "custom" font that uses 2-byte chars encodings to encode all the ASCII characters. This will require me to force type 0 somewhere, or make a font somewhat from scratch... We'll see if I get around to it. Will post again if I do with my findings

 

[mn1712trungson]

Sorry for not telling you how to do it specifically for NintyFont.

When pressing the ADD key. You need to click in the last box to change the code point value.
Only Nintyfont, as far as I know, can create custom character maps.
Tinke was simply determining whether or not the character map was correct.

Finish and save the character map. Load font on Tinke and check character map value.
BTW, the "Roman Text Test!" you show.
It's actually half-width Japanese character set in the end Unicode chart.

And the reason why EU fonts do not appear on Japanese rom is straightforward.
Japanese rom encoding char will read the backwalk hex value (Byte 3-4 and Byte 1-2).
00 54 = T on Big Endian but on NDS it's become 54 00 Little Endian.

If you need some help. I can modify the font (ASCII character map) for you.

Post automatically merged: Nov 19, 2022

I spent some time debugging both games (EN) and JP.
I realized NFTR font in rom acts like a bitmap font + table (encoding in any format is fine).
So rom skip the next offset value and only take width value.
Bitmap font is drawn on RAM and pushed straight to BG1 (like NES or SNES games).

In JP rom, two-byte hex values serve as an opcode (0D - 0A for break lines, 40 = @ + number).

That means there must be a routine that reads the string in 2-byte format before processing the txt in 4-byte format.

They removed the 4-byte routine from EN ROM and processed the opcode + text in 2-byte format as well.
Of course, NFTR encoded Shift-JIS as well (next offset value used). Only change is txt file encodes ASCII.

 

[lps]

You’ll need to decompress overlay9_002 and find font width table:
USA offset: 0x1EF58
EUR offset: 0x1F158
Try to find something similar for JAP rom.

 

[MRJPGames]

Sorry for the late reply

Sorry for not telling you how to do it specifically for NintyFont.

When pressing the ADD key. You need to click in the last box to change the code point value.
Only Nintyfont, as far as I know, can create custom character maps.
Tinke was simply determining whether or not the character map was correct.

View attachment 338526

Finish and save the character map. Load font on Tinke and check character map value.
BTW, the "Roman Text Test!" you show.
It's actually half-width Japanese character set in the end Unicode chart.

View attachment 338527

And the reason why EU fonts do not appear on Japanese rom is straightforward.
Japanese rom encoding char will read the backwalk hex value (Byte 3-4 and Byte 1-2).
00 54 = T on Big Endian but on NDS it's become 54 00 Little Endian.

View attachment 338528

If you need some help. I can modify the font (ASCII character map) for you.

Post automatically merged: Nov 19, 2022

I spent some time debugging both games (EN) and JP.
I realized NFTR font in rom acts like a bitmap font + table (encoding in any format is fine).
So rom skip the next offset value and only take width value.
Bitmap font is drawn on RAM and pushed straight to BG1 (like NES or SNES games).

In JP rom, two-byte hex values serve as an opcode (0D - 0A for break lines, 40 = @ + number).
View attachment 338626
That means there must be a routine that reads the string in 2-byte format before processing the txt in 4-byte format.

They removed the 4-byte routine from EN ROM and processed the opcode + text in 2-byte format as well.
Of course, NFTR encoded Shift-JIS as well (next offset value used). Only change is txt file encodes ASCII.

Sorry for the late reply, life kinda got in the way.

Regardless I have learned a lot about NFTR format, which is very nice. However I think editing the font won't help in the end.
I've been able to add new chars, to modify fonts to "move" ASCII to JASCII/Full Width encoding for them. However whatever I do the game will use static/constant spacing regardless.

Also you said that the mapping "type" as displayed in Tinke, and described here has something to do with proportionality. However it seems to me that it only controls how the character map is read, as described in the documentation of the format on problemkaput's website. So if I'm not mistaken it only controls how that mapping is done, is it a start at this tile and continue for entire range, is it a linear list of the entire range with the tile id's or is it a list of char encoding to tile id. None of which have much to do with proportionality (eg. type 2 will still be treated proportional in other game if I'm not mistaken)

As for the actual encoding/decoding used I think it's just proper shift_jis. 2 byte characters are processed seemingly normally, just not displayed. I believe this because introducing only one of such characters does not cause all characters afterwards to get jumbled up because they are no longer aligned. And as CRLF are done with 0x0D0A, both 2-byte chars, not the 4 bye variants you showed.

From all of this I'm concluding that my original suspicion that proportional width is simply being ignored by the rendering code is most likely correct?

You’ll need to decompress overlay9_002 and find font width table:
USA offset: 0x1EF58
EUR offset: 0x1F158
Try to find something similar for JAP rom.

I'll give it a shot, but my binary executable hacking skills are close to 0 (experience at least). But I'll see if I can get started with those US/EU offsets!