Retroarch Wii U, devkitpro r38 and WUT

tl;dr Devkitpro r38 looks like it will work to build retroarch WiiU with some minor makefile changes. Of course, testing, code review and continuous integration are a whole other matter.

I spent a few hours updating retroarch build scripts from unsupported Devkitpro r29 (per Wii U wiki) to the supported Devkitpro version r38. Before I spend too much more time, can someone help answer these 3 questions.

1. Has someone already updated the build scripts to work on the supported Devkitpro releases (ex r38)? Let me know and I'll stop working on this.

2. If I spend time on this, would it be possible to commit the fixes/updates added to the main branch? (and who signs off on this?... or is that a question to be asked on another forum?)

3. How does testing work for the WiiU retroarch builds? Assuming we have a stable build config with a working build, and an ok for question 2, there are 30+ cores that need testing plus the main binary. I don't think I could do justice to retroarch quality if I tested all by myself.


== Rationale for update ==
Tl;dr - Devkitpro 38 is the supported release and very polished, Devkitpro 29 has no support and is in a rough state. detailed answer: TBH, I'm new to this, but from reading source, posts & wikis plus some mucking about....
1. Devkitpro for Wii U received a significant investment over 2019 & 2020 in the areas of build automation and linking. The new Wii U Toolchain (wut) seems like the only supported linker solution for all Wii U DevkitPPC projects. Installation is very easy and clean.
2. The old build scripts (pre34) suffer from lot of small creeping issues. ex. The Devkitpro r34 release & commit notes imply the old rpx support (incl r29) was incomplete/broken, so retroarch sounds like it's missing out on fixes. I pulled the r29 build scripts from archives, but these didn't compile/work out of the box, and the failures tended to be the uncaught/silent type .
3. IIRC, Libraries like ffmpeg appear to be packaged for WiiU devkitpro r38. (my interest). This means a community is sharing the load of keeping the versions updated, tested and fixed, and such shared WiiU libraries have a home. The ffmepg core is a separate topic, and I am setting my expectations really low. I don't plan on trying this until retroarch fully builds on.

== Changes ==

I'm approaching this with next to no background in retroarch, homebrew, devkitpro or WiiU/ppc development, so apologies if my notes are too detailed for the audience. In case anyone wants to do it themselves, I'm documenting what has worked so far.

First, remove obsoleted LD_FLAGS from all core makefiles then build cores. Second, change the Makefile.WiiU linker targets from old linker libraries (ld, scripts, asms) that are bundled with retroarch to the standard Wii U Toolchain (wut) style targets. Third, we need to deal with the static core linking scripts specific to WiiU, since each core is a separate executable.

[CORES] First part, build the cores without error and successfully generate libretro_xyz_wiiu.a libraries.

* install devkitpro per devkitpro's platform-specific instructions (I used mac os package)

* install PPC and toolchain per instructions
sudo dkp-pacman -Syu devkitPPC wut-tools
export DEVKITPRO=/opt/devkitpro
export DEVKITPPC=/opt/devkitpro/devkitPPC
# my mac version
/opt/devkitpro/devkitPPC/bin/powerpc-eabi-gcc --version
powerpc-eabi-gcc (devkitPPC release 38) 10.2.0

* install extra stuff some of the WiiU cores will need.
% sudo dkp-pacman -Syu ppc-libpng libogc

Make sure cores build first. Checkout per retroarch wiki.
git clone.../libretro-super.git (sorry, Noobs not allowed to post urls)
$ cd libretro-super
$ ./libretro-fetch.sh

Most important step: Find and removed all instances of the -mwup flag. From looking through devpro team posts, wikis, old code submits, It sounds this older (unsupported) approach required projects such as retroarch to copy and ship WiiU linker libraries/scripts with their projects. The new approach is to rely on linker scripts officially owned and supported by Devkitpro, and deployed using package managers. The first part of this migration is ensure the deprecated -mwup flag is stripped. All you need to do is ensure the cores build, and the libretro_xyz_wiiu.a files get compiled without error.

# expect this error out of the box: powerpc-eabi-g++: error: unrecognized command-line option '-mwup'
# small tweak - makefiles for cores "fbneo" & "lutro" seem to need this to locate gnu archiver binary (AR). All makefiles should be fixed to rely on $DEVKITPPC/bin
$ export PATH=$PATH:$DEVKITPPC/bin
# mass change - remove all references of -muwp
$ grep -irl '\-mwup' .
# manually changes the files you want or pipe to sed -i for in place changes. (I did this manually. There are recipes on the web, but Noobs like me can't post urls )
$ ./libretro-build-wiiu.sh

If you made all the changes, this produces an output of "N core(s) successfully processed:" and No "cores failed". If this doesn't work, I don't see much point in continuing.

== Planned =

[Makefile.WiiU] Second part, I'll need to update Makefile.wiiu to conform with modern DevkitPro / wut makefile target format.
* check out retroarch per retroarch (sorry, Noobs can't post URLS, so adding "...". See WiiU retroarch wiki)
git clone .../ /RetroArch.git retroarch
cd retroarch

## this will break with familar linker errors called out by QuarkTheAwesome (libretro/RetroArch/issues/6781 on github com - sorry, Noob's can't post full URLs)
make -f Makefile.wiiu

TBD work, but I'll continue if no-one points me to this being solved already (and time permits)
* Open Makefile.wiiu
* Open devkitPro/wut/tree/master/samples/make (sorry, Noobs can't post URLS. This path is on github com)
* Refactor Makefile.wiiu to conform to example WUT target format.

[Distscripts] Third part (probably)
(TIL) The wiki for WiiU retroarch release notes that retroarch for WiiU is special. Specifically WiiU statically links all core libraries into executables. Other platforms like linux and windows load the cores as dynamic libraries. WiiU handles this extra step through a custom "dist script" solution. TBD - learn work needed to adapt WUT link syntax to dist scripts. Open question: Why does WiiU need distscripts? Would it be cleaner/easier to just rely on libretro-super documentation + an extra makefile build rule?

 

Wow, thanks to everyone for jumping in and spending the time on responses. Its encouraging you folks can build WiiU using DEVKITPPC release 38

I'm very confused about the -mwup option on WiiU builds. When I look ad commit r34 of devkitpro, I see the devkitpro code supporting the -mwup has been completely removed. (below)

@Quincy or @mive : Could I trouble you run the following commands on your environment? Please use the env variable used by the makefiles, not expanded absolute path.

% ${DEVKITPPC}/bin/powerpc-eabi-gcc -mwup -v
EXPECT: many lines, including
...
powerpc-eabi-gcc: error: unrecognized command-line option '-mwup'
...
gcc version 10.2.0 (devkitPPC release 38)

If gcc is the same as mine, and you don't see the -mwup error, then that tells me there's something environmental with my build. If you do get the -mwup error, that probably means there's something strange with the commands I ran off the WiiU retroarch wiki. (ex. ./libretro-build-wiiu.sh ). Perhaps as @Quincy suggested, I should use ( libretro_super.sh from the libretro .

(also, if this is better communicated over "4TU Discord" per @dojafoja , let me know)

EDIT: Oh. I noticed you folks are Advanced and probably have been building retroarch over many years. Just for sanity's sake, how hard would it be for you guys to start with a Noob "Greenfield" environment setup such as I did in the original post? Like, a aompletely fresh install of the tool chains, on an OS that has never had a devkitpro installed before.

EDIT EDIT: Rather than waste your valuable time on building a clean system... Let ME first do a fresh install of Ubuntu with the Devkitpro r38 set up following the Devkitpro and retroarch linux set up wikis. THEN I'll try to build the WiiU release with their latest gcc and supported options..

= On removal mwup =

google search for "devkitPPC release 34 is now available"
Commit in r34 removing -mwup and crtls/wup.ld

A bit more on rpx compilation and packaging: It does look like the rpx tool chain in retroarch is frozen in time (pre-2017). I'll spend a few more hours on at using the versions from the 2021 Devkitpro Wii U Toolchain, but if I get ratholed, I'll take a step back and try a fresh checkout in hopes of replicating what @Quincy and @mive are able to do out of the box.

 

np! thanks for the clarification.

Did I download from here: yes. However, you made me wonder if the retroarch core directories pulled from git are the same release as the core directories pulled from libretro-super. Let me check. Ex.

EDIT: I just created a new vm and am going through code and checkouts. This makes a lot more sense now. The main package makefile has no -mwup option, but the core makefiles do!

Main checkout command
$ git clone .../libretro/RetroArch.git retroarch
RESULT: Only checks out code to build single retro_arch executable.
$ grep -ir '\-mwup` retroarch
[no results]

Core checkout command
$ git clone .../libretro/libretro-super.git
$ cd libretro-super/
$ ./libretro-fetch.sh
PLATFORM: Linux
ARCHITECTURE: x86_64
TARGET: unix
=== RetroArch
Fetching retroarch...
git clone ".../libretro/RetroArch.git" ".../libretro-super/retroarch"
Cloning into '../libretro-super/retroarch'
...
RESULT: checks out retroarch executable (again), dependencies, and all CORE
~/libretro-super$ grep -ir '\-mwup' retroarch
[NO RESULTS]
~/libretro-super$ grep -ir '\-mwup'
libretro-snes9x2010/Makefile.libretro: CFLAGS += -DGEKKO -DHW_RVL -mwup -mcpu=750 -meabi -mhard-float -D__ppc__ -DMSB_FIRST
Binary file libretro-desmume2015/.git/objects/pack/pack-20d32e4fd03224910f93c305637c61b625b6b807.pack matches
.. [Many, many more results]

Observation:
From the libretro_super build errors, approximately half the supported wiiu cores have -mwup in them. Looking at a core makefile that has -mwup in it (./libretro-snes9x2010/Makefile.libretro)
, we see this:
else ifeq ($(platform), wiiu)

To build cores in libretro_super, we need to supply platform=wiiu, such as "platform=wiiu ./libretro-super.sh" to set $(platform) Setting this would force the -mwup to be used for at least half the wiiu cores (thus the complete WiiU core batch build command would fail using devkitpro r38).

Speculation 1: I should not be building the cores pulled by libretro-super or using the libretro-super script, but something else?
Speculation 2: core makefiles that are configured to build for wiiu, but still have -mwup, are not supported? ... or I shouldn't equate WiiU supported cores to cores that have this WiiU specific build statement like "else ifeq ($(platform), wiiu)"
Speculation 3: Cores that build for WiiU are NOT using WiiU build options and compiler flags. If you set the platform=wiiu variable on libretro-super, -mwup WOULD be used along side other WiiU options. (for at least half the cores at least). This might be ok; building a PPC executable on linux using linux core build options may just work on the WiiU. The makefile execution is just not what a reader would expect. (ie we're not using " else ifeq ($(platform), wiiu)" build commands for wiiu builds)

 

Thanks @QuarkTheAwesome - you made me dig even deeper. I see you're a committer to both Makefile.wiiu and author of WUT. cool.

One question, when you made changes to Makefile.wiiu, do you happen to remember what devkitpro environment you used? (ex. version, was it packaged by pacman, etc)

Also, fwiw, only one Makefile option builds for Makefile.Wiiu. I suspect there is more amiss than just the build environment, and some hint how to fix it.

== Works - SALAMANDER ==
In Makefile.wiu
GRIFFIN_BUILD = 0
SALAMANDER_BUILD = 1

make -f Makefile.wiiu
AS wiiu/system/stubs_rpl.S
LD objs/wiiu-salamander/retroarch_wiiu_salamander.rpx.elf
ELF2RPL objs/wiiu-salamander/retroarch_wiiu_salamander.rpx
cp objs/wiiu-salamander/retroarch_wiiu_salamander.rpx retroarch_wiiu_salamander.rpx
CC wiiu/system/dynamic.c
AS wiiu/system/stubs_elf.S
LD objs/wiiu-salamander/retroarch_wiiu_salamander.elf

cp objs/wiiu-salamander/retroarch_wiiu_salamander.elf retroarch_wiiu_salamander.elf

== Fails - default ==
Leave Makefile.wiiu unmodified
GRIFFIN_BUILD = 0
SALAMANDER_BUILD = 0
make -f Makefile.wiiu
..
LD objs/wiiu/retroarch_wiiu.rpx.elf
/opt/devkitpro/devkitPPC/bin/../lib/gcc/powerpc-eabi/10.2.0/../../../../powerpc-eabi/bin/ld: objs/wiiu/retroarch.o: in function `command_event':
retroarch.c....): undefined reference to `retro_set_environment'

...

== Fails - GRIFFIN ==
Edit makefile.wiiu, set to GRIFFIN
GRIFFIN_BUILD = 1
SALAMANDER_BUILD = 0
make -f Makefile.wiiu
...

CC griffin/griffin.c
In file included from griffin/griffin.c:353:
griffin/../gfx/video_shader_parse.c: In function 'video_shader_load_preset_into_shader':
griffin/../gfx/video_shader_parse.c:1605:8: warning: unused variable 'reference_depth' [-Wunused-variable]
1605 | int reference_depth = 1;
| ^~~~~~~~~~~~~~~
In file included from griffin/griffin.c:716:
griffin/../input/drivers_joypad/wiiu/wpad_driver.c: At top level:
griffin/../input/drivers_joypad/wiiu/wpad_driver.c:39:12: error: conflicting types for 'channel_slot_map'
39 | static int channel_slot_map[WIIU_GAMEPAD_CHANNELS] = { WPAD_INVALID_CHANNEL, WPAD_INVALID_CHANNEL };
| ^~~~~~~~~~~~~~~~

 

@QuarkTheAwesome and @mive Thanks! So simple. The entire build works for me with default configs.

To make sure this is reproducible, I'm going to nuke my setup and start from scratch. Afterwards, I'll test a few rpx's before messing with wut changes.

Steps to rebuild
1. GITHUBCOM = github repo url, but Noobs can't post links
2. I'll edit this if I find mistakes tomorrow (will take hours)
3. note to others: the wiiu wiki steps are totally correct, but not listed in the chronological order you see below

BASE=`pwd` # like your homedir

export DEVKITPRO=/opt/devkitpro
export DEVKITPPC=/opt/devkitpro/devkitPPC
# Hack for devkitpro r38 - TBD fix Makefiles that don't set this
export PATH=$PATH:$DEVKITPPC/bin

# FIRST - BUILD CORES
# check out libretro cores using libretro-super
cd $BASE
git clone GITHUBCOM /libretro/libretro-super.git
cd $BASE/libretro-super
./libretro-fetch.sh

# Hack for devkitpro r38 - TBD make a diff to clean up options for cores makefiles
#for now, simply in place remove all -mwup references under libretro-super
cd $BASE
grep -rl '\-mwup' --include \?akefile\* libretro-super 2> /dev/null | xargs -I{} sed -i '' 's/\-mwup//' {}

# build all cores
cd $BASE/libretro-super
./libretro-build-wiiu.sh

# Bask in the cores that were compiled
ls $BASE/libretro-super/dist/wiiu

# FRONTEND & RPX's SECOND
# Check out retroarch to build "retroarch" frontend
cd $BASE
git clone GITHUBCOM /libretro/RetroArch.git retroarch

# Copy over a single core to distscripts, name it libretro_wiiu.a
cp $BASE/libretro-super/dist/wiiu/snes9x2010_libretro_wiiu.a $BASE/retroarch/dist-scripts/libretro_wiiu.a

# Copy over ALL cores you want to package
cp $BASE/libretro-super/dist/wiiu/*.a $BASE/retroarch/dist-scripts

# Make the "retroarch" frontend elf and rpx
cd $BASE/retroarch/dist-scripts
./wiiu-cores.sh

# See the resulting build
cd $BASE/retroarch && find pkg/wiiu

 

Not so great news. Launching the newly built retroarch.rpx resulted in the error "There is a problem with system memory".

For now, I am ignoring the cores, and just attempting successfully load the retroarch.rpx frontend. Other retroarch posts imply this is a runtime memory size issue. If so, I'll look more into the WiiU retroarch compiler/link options to see if this is something I can optimize. Any and all suggestions appreciated

Testing:
Preparation: create new Wii U HBL image on new SD card. confirm HBL boots.

Test 1: Download latest official retroarch release. Copy to SD card per instructions.
ls -l Downloads/RetroArch_rpx/wiiu/apps/retroarch
[email protected] 1 testuser staff 5077 Aug 11 14:29 icon.png
[email protected] 1 testuser staff 529 Aug 11 14:29 meta.xml
[email protected] 1 testuser staff 145980 Aug 11 14:29 retroarch.rpx
Insert SD card into WiiU, power on. Open HBL on WiiU. Select Retroarch app.
Result 1: Retroach frontend loads after 20sec. Snes2010 core loads and returns to the Retroach frontend UI.

Preparation: rm -f SD/wiiu/apps/retroarch SD/retroarch
Test 2: Copy build from pkg/wiiu to SD card (both ./wiiu/apps/retroach and ./retroarch/cores.)
use pkg version
% find retroarch -name 'retroarch*.rpx'
retroarch/objs/wiiu-salamander/retroarch_wiiu_salamander.rpx
retroarch/objs/wiiu/retroarch_wiiu.rpx
retroarch/pkg/wiiu/rpx/wiiu/apps/retroarch/retroarch.rpx
Use the pkg version.
ls -l retroarch/pkg/wiiu/rpx/wiiu/apps/retroarch
total 336
-rw-r--r-- 1 testuser staff 5077 Jan 21 13:40 icon.png
-rw-r--r-- 1 testuser staff 529 Jan 21 13:40 meta.xml
-rw-r--r-- 1 testuser staff 157820 Jan 21 13:40 retroarch.rpx
Insert SD card into WiiU, power on. Open HBL on WiiU. Select Retroarch app.
Result 2: After several seconds, the WiiU shows the error screen "There is a problem with system memory",