Most custom compression formats are minor variations on the standard theme.
Generally every few bytes (distance can vary but will typically be one nicer for the hardware to handle -- most things prefer multiples of 8, 16 or the like rather than 13 or whatever) will be a flag saying either move along as there is nothing to see or there is another flag that is split between how many bytes to go backwards from this point to find what you want and how many to read once you get there*.
Any customisation there will break a basic decompression tool but if you have source to it (like cue's and dsdecmp will) you can change them easily enough.
*note that the bits you get for this value to read might need a +2 or however long the flag is as you don't want to have your compression increase the file size if you can help it (going back 30 bytes to read 0 being pointless after all, indeed worse as you have increased the size of the file for no gain).
You can also read the source code for cue's stuff
https://www.romhacking.net/utilities/826/ or
https://github.com/barubary/dsdecmp if you want.
http://members.iinet.net.au/~freeaxs/gbacomp/#Image Compression on GBA and
http://problemkaputt.de/gbatek.htm#biosdecompressionfunctions
Finding out what the format is can vary.
You can try to make sense of it manually by looking at the file itself. Doable enough but tedious. I prefer text for this approach myself (at the start with nothing to reference the text starts out fairly clear/plain and then increasingly turns to gibberish as you get more and more repeats) but graphics are also kind of an option (they also start out OK but get worse as the compression goes on).
If you can find out where the file is loaded in the game (say by swapping the files noted above around and seeing what changed in the game, maybe deleting them/blanking at first to see if something crashes or errors out) and dump the memory. Things tend to be held as decompressed in the memory so you now have an example of the compressed file and decompressed file.
Referencing back and forth between the two is then far nicer to figure out the encodings used.
Finally (assuming there is no other tool out there -- we have found zip, tar and .gz files in DS ROMs before, though usually stuff the developers left in there and forgot to remove when building the ROM) there is the assembly approach where you find the file being decompressed and follow along with what it does. It is not the hardest assembly out there but you will probably have several steps and have to figure out what the greater purpose of it all is.
In normal DS compression/BIOS compression/BIOS compatible compression the file will start with 10 (older games and some newer ones), 11 (newer games) or 40 (originally I think it was Golden Sun but made it to a few others in even later ROMs) and then be followed by how big the file should be, and that arrangement is what it seems a lot of compression searching tools look for and I was attempting to spot above.
If you have source you have recompression options as well but in a pinch you can continue assuming it is compressed but leave it uncompressed and insert the "nothing to see here" bytes throughout the file. In some cases the compression will be detected/noted in some other file and you can fiddle with that instead (the DS binaries, which use a subtly different take working from the back of the file, BLZ, have this).