Circuit Python on the FOMU in WSL (plus the pains of building your own RISC-V toolchain)

I started down the road of learning FPGA Circuit Python last month, with the (admittedly overly-ambitious) goal of getting Circuit Python working on the ULX3S. I was inspired to revisit this today when I saw Adafruit Blog - CircuitPython 5.2.0 Released - that specifically mentioned LiteX for Circuit Python!! In this case, for the FOMU.

TL;DR - WSL is by default NOT Case Sensitive. Use fsutil.exe in a DOS admin window to change this for each directory. To compile, riscv64-unknown-elf-gcc is needed, but the RISC-V GitHub GNU Repo does not work here to build Circuit Python. Use the FOMU toolchain. The make parameter should be lower case: make BOARD=fomu

If you are interested in Circuit Python, check out this CircuitBrains Deluxe on Crowd Supply.

See Adafruit instructions on building Circuit Python. All of the downloads for Circuit Python are available online. Well, all but of course the FOMU that I am most interested in. Hopefully it will be there soon, so for now we are doing this the hard way. (no fun if it is too easy, right?)

I could not find any documentation on the FOMU build, so I had to dig and search for keywords. The initial part of fetching from GitHub is pretty standard, so we'll start there:

git clone https://github.com/adafruit/circuitpython.git
cd circuitpython
git submodule sync
git submodule update --init
make -C mpy-cross

When doing the git submodule update --init in DOS (I sometimes do that, since my git clones are in C:\workspace - even though I compile in WSL from /mnt/c/workspace/)... I saw these errors:

git submodule update --init
Submodule 'extmod/ulab' (https://github.com/v923z/micropython-ulab/) registered for path 'extmod/ulab'
...etc
Submodule path 'lib/nrfutil': checked out '9e7dfb28a5c6f3d7a19340971b32e0c2b4128ecf'
Submodule path 'lib/tinyusb': checked out '1f95f439e11f519e69d75a4a8b7b9f28eaf5060e'
error: unable to create file tests/decomp-bad-inputs/00/id:000000,sig:11,src:000000,op:flip1,pos:10: Invalid argument
...etc
error: unable to create file tests/decomp-bad-inputs/00/id:000012,sig:11,src:000000,op:arith8,pos:11,val:+6: Invalid argument
error: unable to create file tests/decomp-bad-inputs/00/id:000013,sig:11,src:000000,op:arith8,pos:12,val:-9: Invalid argument
error: unable to create file tests/decomp-bad-inputs/00/id:000014,sig:11,src:000000,op:havoc,rep:16: Invalid argument
error: unable to create file tests/decomp-bad-inputs/00/id:000015,sig:11,src:000000,op:havoc,rep:2: Invalid argument
... many, many more similar errors

When doing the git submodule update --init from WSL just a few moments later, I did not see those errors. Perhaps there's some issue with Windows file names. (edit: I learned later this is likely a case sensitivity issue, see below).

For the FOMU, there's a Makefile in ./ports/litex

cd ./ports/litex
make BOARD=fomu

Edit: note that a case insensitive directory setting may allow BOARD=FOMU to work. ymmv. The safest one to use is a lower-case: BOARD=fomu.

If you see an error like this:

xargs: riscv64-unknown-elf-gcc: No such file or directory
../../py/mkrules.mk:81: recipe for target 'build-FOMU/genhdr/qstr.i.last' failed
make: *** [build-FOMU/genhdr/qstr.i.last] Error 127
make: *** Deleting file 'build-FOMU/genhdr/qstr.i.last'

Then try riscv64-unknown-elf-gcc --version to see if this RISC-V compiler is installed (perhaps you need to install the compiler). sadly, this is a different version of the compiler than is used in my ULX3S toolchain builder, that uses rv32i installed to /opt/riscv32i. Of course it does. Sometimes I wonder if this is all a hard drive manufacturer conspiracy to use as much disk space as possible. ;)

If like me you don't have the right compiler installed, make sure you have plenty of disk space:

This process will start by downloading about 200 MiB of upstream sources, then will patch, build, and install the toolchain. If a local cache of the upstream sources exists in $(DISTDIR), it will be used; the default location is /var/cache/distfiles. Your computer will need about 8 GiB of disk space to complete the process.

Here's what I did in WSL:

sudo apt-get install autoconf automake autotools-dev curl python3 libmpc-dev libmpfr-dev libgmp-dev gawk build-essential bison flex texinfo gperf libtool patchutils bc zlib1g-dev libexpat-dev
git clone --recursive https://github.com/riscv/riscv-gnu-toolchain  
cd riscv-gnu-toolchain  
./configure --prefix=/opt/riscv
sudo mkdir /opt/riscv
sudo chown $USER /opt/riscv
if [ "$(echo $PATH | grep  /opt/riscv/bin)" == "" ]; then export PATH=$PATH:/opt/riscv/bin; fi
make

You may wish to put the export PATH=$PATH:/opt/riscv/bin in your ./bash.rc file.

Do NOT run make linux if you want riscv64-unknown-elf-gcc files! The linux option creates riscv64-unknown-linux-gnu files, not elf!.

My first attempt at install failed. After hours of downloading and building, the errors started here (and yes, I also incorrectly specified linux):

mv -f /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/rtld-libc.aT /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/rtld-libc.a
make[4]: Leaving directory '/mnt/c/workspace/riscv-gnu-toolchain/riscv-glibc/elf'
riscv64-unknown-linux-gnu-gcc    -nostdlib -nostartfiles -r -o /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/librtld.os '-Wl,-(' /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/dl-allobjs.os /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/rtld-libc.a -lgcc '-Wl,-)' \
          -Wl,-Map,/mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/librtld.os.map
riscv64-unknown-linux-gnu-gcc    -nostdlib -nostartfiles -shared -o /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/ld.so.new               \
          -Wl,-z,combreloc -Wl,-z,relro -Wl,--hash-style=both -Wl,-z,defs       \
          /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/librtld.os -Wl,--version-script=/mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/ld.map            \
          -Wl,-soname=ld-linux-riscv64-lp64d.so.1                       \
          -Wl,-defsym=_begin=0
/opt/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld: /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/librtld.os: in function `process_envvars':
/mnt/c/workspace/riscv-gnu-toolchain/riscv-glibc/elf/rtld.c:2484: undefined reference to `__GI___open64_nocancel'
/opt/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld: /mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/librtld.os: in function `dl_main':
/mnt/c/workspace/riscv-gnu-toolchain/riscv-glibc/elf/rtld.c:1317: undefined reference to `__access'
/opt/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld: /mnt/c/workspace/riscv-gnu-toolchain/riscv-glibc/elf/rtld.c:2103: undefined reference to `__access'
/opt/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld: /mnt/c/workspace/riscv-gnu-toolchain/riscv-glibc/elf/rtld.c:1564: undefined reference to `__GI___close_nocancel'

and eventually culminated here:

/opt/riscv/lib/gcc/riscv64-unknown-linux-gnu/9.2.0/../../../../riscv64-unknown-linux-gnu/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
Makefile:496: recipe for target '/mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/ld.so' failed
make[3]: *** [/mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d/elf/ld.so] Error 1
make[3]: Leaving directory '/mnt/c/workspace/riscv-gnu-toolchain/riscv-glibc/elf'
Makefile:258: recipe for target 'elf/subdir_lib' failed
make[2]: *** [elf/subdir_lib] Error 2
make[2]: Leaving directory '/mnt/c/workspace/riscv-gnu-toolchain/riscv-glibc'
Makefile:9: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/mnt/c/workspace/riscv-gnu-toolchain/build-glibc-linux-rv64imafdc-lp64d'
Makefile:234: recipe for target 'stamps/build-glibc-linux-rv64imafdc-lp64d' failed
make: *** [stamps/build-glibc-linux-rv64imafdc-lp64d] Error 2

I was about to open a GitHub issue, and then thought - hm... perhaps I just read the rest of the README, eh? I know, I know, actually RTFM'ing takes away all the fun, but it is rather embarrassing when someone tells you to read the fine manual in response to asking for help. Sure enough, I found this:

If building a linux toolchain on a MacOS system, or on a Windows system using the Linux subsystem or cygwin, you must ensure that the filesystem is case-sensitive. A build on a case-insensitive filesystem will fail when building glibc because *.os and *.oS files will clobber each other during the build eventually resulting in confusing link errors. @tnt aka smunaut has a RISC-V USB project in the works.

In order to install RISC-V for WSL I found this article that noted after Windows 10 version 17093 there's a command fsutil.exe for setting case sensitivity:

fsutil.exe file setCaseSensitiveInfo riscv-gnu-toolchain enable

Note this needs to be run in a DOS Window, not WSL, and with Administrative permissions:

BUT! Note the command is only for the specified directory, and the attribute is NOT inherited  (and certainly does not get applied to each existing sub-directory). So we need to iterate through all sub-directories individually. Yes, I'm sure many readers are rolling their eyes at this point, asking why I am doing this in WSL and not a "real" Linux machine or even a VM; what am I doing in Windows? I ask myself that sometimes, too. Well, I just think WSL is just too cool! Plus, I suppose that I like the challenge. So anyway, here's what I did to set case sensitivity:

cd \workspace\riscv-gnu-toolchain\
for /f %f in ('dir /ad /s /b') do fsutil.exe file setCaseSensitiveInfo %f enable

There are many directories. Be patient...

As a reminder, that command is meant to be executed interactively. If placed into a batch file, the percent symbols need to be doubled up.

Now that everything in the riscv-gnu-toolchain directory is case sensitive, I ran make again:

cd /mnt/c/workspace/riscv-gnu-toolchain
make

This does a full rebuild, taking again a very long time. Did I mention being patient?

If you can run /opt/riscv/bin/riscv64-unknown-elf-gcc --version but see a lot of errors when doing a make BOARD=fomu like:

/opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/include/stdint.h:9:16: fatal error: stdint.h: No such file or directory

That means the RISC-V make either failed... or you didn't wait until it finished. (for me, it was the latter; did I mention being patient?) Once the RISC-V make is complete, the FOMU code can be built:

cd /mnt/c/workspace/circuitpython/ports/litex
make BOARD=FOMU

Which resulted in this output:

$ make BOARD=FOMU
Use make V=1, make V=2 or set BUILD_VERBOSE similarly in your environment to increase build verbosity.
QSTR updated
make: msgfmt: Command not found
../../py/py.mk:338: recipe for target 'build-FOMU/genhdr/en_US.mo' failed
make: *** [build-FOMU/genhdr/en_US.mo] Error 127

so ok, a quick google yields something useful on stackoverflow

apt-get install gettext

and so even after than, I saw even more errors like this:

/opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: build-FOMU/lib/libm/math.o: in function `floorf':
/mnt/c/workspace/circuitpython/ports/litex/../../lib/libm/math.c:790: undefined reference to `__addsf3'
/opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: /mnt/c/workspace/circuitpython/ports/litex/../../lib/libm/math.c:795: undefined reference to `__addsf3'
/opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: build-FOMU/lib/libm/math.o:/mnt/c/workspace/circuitpython/ports/litex/../../lib/libm/math.c:817: more undefined references to `__addsf3' follow
/opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: build-FOMU/firmware.elf(.text): relocation ".L15+0x0 (type R_RISCV_RVC_JUMP)" goes out of range
/opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: /opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/libgcc.a(divsf3.o): file class ELFCLASS64 incompatible with ELFCLASS32
/opt/riscv/lib/gcc/riscv64-unknown-elf/9.2.0/../../../../riscv64-unknown-elf/bin/ld: final link failed: file in wrong format
collect2: error: ld returned 1 exit status
Makefile:177: recipe for target 'build-FOMU/firmware.elf' failed
make: *** [build-FOMU/firmware.elf] Error 1

So I did another one of these, this time for the circuitpython directory:

for /f %f in ('dir /ad /s /b') do fsutil.exe file setCaseSensitiveInfo %f enable

and things got even worse:

$ make BOARD=FOMU
Makefile:30: *** Invalid BOARD specified.  Stop.

So I tried to clean it, same error:

$ make BOARD=FOMU clean
Makefile:30: *** Invalid BOARD specified.  Stop.

Turns out that was case sensitive, so then it is:

make BOARD=fomu clean
make BOARD=fomu

Ok, at the point, readers must really, really wonder why I use Windows. I know, I know. But I'm persistent.

It still failed. I gave up and asked for help.
I didn't get an answer to "should I use the FOMU toolchain instead of the RISC-V, so I thought I'd try it:

# fetch and extract
wget https://github.com/im-tomu/fomu-toolchain/releases/download/v1.5.6/fomu-toolchain-linux_x86_64-v1.5.6.tar.gz 
tar -xzf fomu-toolchain-linux_x86_64-v1.5.6.tar.gz
sudo mv fomu-toolchain-linux_x86_64-v1.5.6 /opt/fomu-toolchain-linux_x86_64-v1.5.6

# set the path
if [ "$(echo $PATH | grep fomu-toolchain-linux_x86_64-v1.5.6)" == "" ]; then export PATH=$PATH:/opt/fomu-toolchain-linux_x86_64-v1.5.6/bin; fi

# verify it is in the path:
which riscv64-unknown-elf-gcc
riscv64-unknown-elf-gcc --version

cd /mnt/c/workspace/circuitpython/ports/litex
make BOARD=fomu  clean

And... TADA! A successful Circuit Python build for the FPGA FOMU!! Whew.

gojimmypi:/mnt/c/workspace/circuitpython/ports/litex
$ make BOARD=fomu
Use make V=1, make V=2 or set BUILD_VERBOSE similarly in your environment to increase build verbosity.
/opt/fomu-toolchain-linux_x86_64-v1.5.6/bin/../lib/gcc/riscv64-unknown-elf/8.3.0/../../../../riscv64-unknown-elf/bin/ld: warning: section `.bss' type changed to PROGBITS

739752 bytes free in flash firmware space out of 1048576 bytes (1024.0kB).
116920 bytes free in ram for stack and heap out of 131072 bytes (128.0kB).

Create build-fomu/firmware.dfu
python3 ../../tools/dfu.py -b build-fomu/firmware.bin -D 0x1209:0x5bf0 "build-fomu/firmware.dfu"

Other notes (thanks @pdp7): Using Python for creating hardware

Migen Step by Step Tutorial

LiteX is a fork of MiSoC

MiSoc and LiteX are both based on Migen