If it's the message "invalid settings" or something like that, this means you did not provide a correct WEP key:
A 40 bit key with 5 ASCII-chars. It can be converted to 10 hexadecimal values.
A 104 bit Key with 13 ASCII-chars. It can be converted to 26 hexadecimal values.
Some devices allow a password of arbitrary length similar to the PSK in modern WPA(2). This password is somehow converted to the hexadecimal key (md5-like hash or something). I could not get this to work. Even with a second computer. [But… why should someone want to use it on a computer? WEP is – as we all should know by now – insecure and basically useless.]
The DS accepts the key both, in hexadecimal form, and as ASCII-chars – but it shows 32 input fields. While Wikipedia claims that 24 bit are used as an “initialization vector”. So why is it called WEP-128 but WEP-40? Why does the DS have 32 (=128Bit with hex-values) input fields? It even accepts a key with 32 Hex-values (or 16 ASCII-chars). No idea.
But I got the DS successfully connected (to a test hotspot on a live Ubuntu) with every other option:
5/13 ASCII and 10/26 Hex