Wall o' Text incoming; here is everything I've learned from editing animals in New Leaf:
Naturally the easiest sort of edit is to just replace one villager for another. Villagers are stored in NPC/Normal/Model, others in NPC/Special/Model. As has already been indicated, not all special models will function well as villagers. The easiest way to ID the models is to open them in Ohana and look at the textures. Just copy the model you want into the Normal/Model folder and rename it to overwrite the villager you're replacing. Alternatively you can modify an existing villager by extracting their textures in Ohana and reimporting them after changes have been made. Lastly, you can extract textures of animals from HHD and import them onto the same style model in NL. As has been stated the model formats are different so this is necessary. Using standard Ohana, you have to open the HHD villagers as Models, then go to the textures tab and extract. Open the NL models right in the textures tab.
However, it's not as much fun to have a custom villager that still is named the original, etc. So here are a few other edits you can do to "clean up" your edit. Many of these involve text editing, so I'll point you to this previous post of mine as to how the text strings are stored:
Animal Crossing: New Leaf ROM Hacking. The post is about name editing but the principles apply to any string file.
1. Name
Villagers names are found in Script/Str/STR_NNpc_name.umbst
Special character names are found in Script/Str/STR_SPNpc_name.umbst
I have not tried tampering with them, but my guess is that the _ASR files of each play into the megaphone, since they contain phonetic versions of some of the names?
2. Default catchphrase
Found in Script/Str/STR_NNpc_habit.umbst
3. Personal Info
STR_NNpc_dream, STR_NNpc_family and STR_NNpc_skill all deal with elements used on April Fools' Day. Since many of these strings are shared among the animals, editing any strings here will change it for any of the animals that use it. I have no idea where it determines which animals use which strings.
4. Pic
To edit your new villager's pic, find first the model of it in the Item/Model folder as brm_***##.bcres. Change the one texture to what you want. The "motto" for the back of the picture is stored in Script/Talk/SYS_Motto.umbst. (This particular folder contains all the basic dialogue for the game as well, in case you're feeling super ambitious). You also will need to edit Script/Str/STR_Item_name.umbst to have its name display properly in inventory.
5. Defaults
There are two files in NPC/Normal that control much of the default info for each villager. ConstPack.bin contains a great deal of default settings that I have not taken the time to fully explore. However, what I can say is that it consists of packs of 34 bytes that define all the data for a given animal. Set HxD to a width of 34 to view it easily. The first 2 bytes seem to be simply a byte-reversed increment. Obviously bytes 10-15 (0A-0E) are animal's five character id. The rest I have not dug into; I'm guessing this may be where birthdays are stored, definitely one of these is personality. However, noting where an animal falls in this list is essential to help find their info in the other file, Package.bin.
The "packs" in this file are 88 bytes long and contain all the item defaults for the animal, very similar to how their info is stored at their entry in garden.dat when they move in. For example, the first 2 bytes are the song that plays in their house. Also here is their umbrella, shirt, and default house furniture. The order for the animals is the same as ConstPack.bin. Alternatively, you could search for items you know a particular animal has by default.
The layout of an animal's house seems to be set to a few standards that may be set by the ConstPack.bin per animal. Each animal has up to 16 slots for furniture, at least 2 of which are wall items and not all of which will be actually used based on the layout setting. Basically what this means is that with the understanding we have now, you can replace furniture defaults with a similar size item, but we cannot simply set up the room however we want. At any rate, editing this info is identical to editing it in a RAM edit, so see the relevant google docs spreadsheet for item info there.
6. Moving in
This is tricky. Some of the stuff you put in the ROMFS will always be used. Appearance, name, etc. Things like furniture, catchphrase and almost anything else "changeable" about the animal will only reflect your edits if they are freshly moved in by the game itself after you edit. Replacing them by save game editor will not work the same. Editing an existing villager in your town won't do it. However, since those are the easiest ways to bring in an animal you want, you can set the things you want (catchphrase, furniture, etc) by hex editing your save file instead.
And that's that. The trickiest part is getting a handle on editing the string files. If you need to adjust a lot of offsets at once, some excel string manipulation formulas and some dec2hex/hex2dec magic might make that much easier to automate.
I'll do my best to follow up to questions, comments and witty remarks.