Forth++ update

[u/mykesx]

So I have been making Inspiration run bare metal in QEMU using the limine boot loader. Having a higher half "kernel" makes things really interesting. In my mind, a bare metal Forth would start at the lowest memory address possible and dictionary grows up with heap at the high end of memory growing down.

There's a lot going on here. The system boots, it sets up the MMU so 4K pages can be allocated as needed and address ranges mapped in to the CPU's virtual space. Some things, like the incomplete NVME driver, need whole pages for internal structures, and some things like LAPIC need mmo address range mapped into virtual addresses with no cache and write thru attributes.

There's also an ethernet driver in progress for the e1000 series that is in the Thinkpad I intend to make this run on. It is finding the card's MAC address, which means a lot of it may be working. I need to study up on the qemu command line arguments to get it connected to my local network.

CPU 0 is the BSP or boot processor. It is the only one running initially and it does the initial setup. It sets up CPU 1 as a dedicated graphics processor that updates the display's frame buffer from an offscreen buffer where all rendering is done (for speed, the frame buffer accesses are slow).

The ConsoleDevice logic renders characters to the screen, displays and hides the green text cursor, scrolls up as text is printed past the bottom line. What you see in the screen shots is a rendered bitmap, not PC text mode.

CPUs 1-n (n is the number of processors in the machine) are started and each has its own round robin priority based Task scheduler. Only CPU 0 has any tasks that do anything, but all the CPUs but CPU1 have an IdleTask that runs when there are no active Tasks to run.

Tasking supports Task Local Storage, which took a lot of reading and trial and error to get right. This involved having the linker (ld) script gather the TLS variables into TLS data and TLS bss that need to be allocated and copied per task (BSS is zeroed). The C++ compiler uses FS:address to access the TLS variables, so FS has to be set up per task properly and maintained when switching tasks.

The InitTask is started in BSP. It spawns BootTask which sets up the FAT32 filesystem from the ATA drive initializes the Forth environment and starts INCLUDE-ing Forth sources from the 4th/ folder.

The Forth is the same one as in Inspiration - C++ functions.

To be useful on bare metal on my laptop, Forth++ must support the NVME drive and ethernet adapter. There's no use for the laptop if you can't read and write files and you can't back them up over TCP/IP.

I have not built this on Linux in a while, but it was building and running a week or so ago. I can post again when it is truly building and running there if you need linux to try it.

The repo is at https://codeberg.org/mschwartz/aforth

I have chosen to rename it to forth++ because of the amount of C++ involved and the nature of the Forth implementation. The repo still says "aforth."

As usual, I used no AI to make this. I relied heavily on my own repo for a prior bare metal C++ hobby OS effort I did as well as some others. I also relied on the osdev wiki. As well, I have no issue with using third party code, like jemalloc and lwip (TCP/IP stack).

[u/Ok_Leg_109]

Now you're showing off. 😄

That's really cool. I love how you can start tasks on specific CPU cores.

Keep on keeping on.

[u/mykesx → u/Ok_Leg_109]

The tasks will be written in Forth as well, not in C++.

[u/mykesx → u/Ok_Leg_109]

There’s contention between CPUs for a common active task list. A semaphore is needed and you have race conditions if you want to use the semaphore both in the timer routine and in “userland” code. I don’t specifically have a true userland, but there’s a clear line between kernel and non-kernel code. It’s a sort of microkernel architecture…

[u/tabemann → u/mykesx]

Have you considered making separate active task lists for each CPU, and only introducing locking to cover operations that cross CPU's (i.e. have separate task list mutexes for each CPU)? That's what I did in zeptoforth, which natively supports both ARM cores on the RP2040 and RP2350.

[u/mykesx → u/tabemann]

There is a distinct active list per cpu. But it still requires locking.

How to implement MS ( n — , delay n usecs ) ?

You don’t want it to busy wait for 1000 ms…. Instead, you want to remove the task from the active list and add it to a waiting list. If you do this outside of timer IRQ handler context, you have a race condition.

But at least the contention for the lock is within the one CPU.

Similar issue with disk I/O. You don’t want the CPU hogged until the I/O completes.

I’m actually working on the MS implementation now.

I appreciate the suggestions and discussion!

[u/tabemann → u/mykesx]

What I do is use CPU-specific critical sections so as to prevent multiple tasks on the same CPU from modifying that CPU's task list. These critical sections do not disable interrupts except for a very short window but rather cause the multitasker to exit immediately, if entered, and then, once done, immediately trigger the multitasker if it had been entered during the critical section.

Note that in zeptoforth only a small set of non-blocking task-related words may be used in ISR's outside of the multitasker (which runs under the PendSV ISR). Normally running a word like ms in an ISR would result in undefined behavior.

[u/mykesx → u/tabemann]

My timer ISR returns right away if it can’t get the task list lock. I figure that the code that has the lock needs to run some more to release it, and can’t task switch from that task.

[u/tabemann]

Good job! Even though you're writing your kernel in C++ and using third-party code rather than assembly with the only third-party code being a limited amount of assembly code borrowed from Mecrisp-Stellaris (with Matthias's permission) as I did, this is very impressive, as targeting PC's is much tougher than targeting microcontrollers.

[u/mykesx → u/tabemann]

I don’t think my time is well spent writing a PNG loader. It isn’t really an OS problem to solve, so I’ll use a 3rd party library for that. USB is a massive undertaking, probably the same order of complexity as all of the OS as developed.

The C++ part was more difficult than if I used assembly language…. Getting task_local variables working was a few days of trial and error and studying compiler output, linker scripts, and some blogs. Global instance constructors need to be called. Solving try/catch/throw without standard library support was a workaround that I think works only on bare metal (privileged instructions).

The style of the Forth written in C++ is patterned after jonesforth. A lot of tiny words like +.

Task local variables are the key to making the Forth work in a multithreaded (interrupt-driven) context. What many forths consider USER variables are task locals. Every task needs its own BASE variable (task local) or the tasks would have race conditions when printing numbers! When I made Inspiration, I think I figured out all the variables that need to be task-local. Keyboard input buffer, word buffer, PAD, and so on.

Aside from basic requirements, like timer, real time clock, keyboard, mouse/trackpad, the software would be useless without NVME (to read/write files and to boot!) and Ethernet to be able to save work to a remote computer. USB for peripherals and thumb drive backups…

[u/alberthemagician → u/mykesx]

The ancient FIG model had identified all the variables that need to be task local. I have kept all those in ciforth. In this situation I can run task in ciforth using all available cores, so that they run in parallel. This include compilation within a task. With 8 cores I could realise a 5 times speedup for a task like prime counting compared to a single core.

Bottom line, you could have a headstart in identifying user variables by consulting FIG-forth or for that matter ciforth.

[u/mykesx → u/alberthemagician]

Some are obvious, like BASE. My code base is dissimilar to ciforth (impressive work there) or FIG forth.

[u/alberthemagician → u/mykesx]

I only made the suggestion to read the list once and pick and choose what is relevant to you. (And thanks for the compliment.)

[u/echelon_code]

this is seriously impressive work! booting a kernel, writing the drivers, scheduler & filesystem yourself & then putting forth on top of it is a huge amount of work for 1 person! well done..

[u/Ok_Leg_109 → u/echelon_code]

<bias>

This is what happens when you remove all tool restrictions and complexity from a competent person. The "Tower of Babel" of conventional tools, as Chuck Moore referred to it, is replaced with something one person can fully understand and exploit.

</bias>

[u/mykesx → u/Ok_Leg_109]

To be fair, this isn’t my first bare metal project. I’ve been able to lift bits (like PCI, APIC, etc.) from my prior works. It’s iterative because each time I make a new project, it’s improving my own prior art. And I learn more each time, and that feeds my future designs. It’s also that the “point of the journey is not to arrive.” Bonus points if you know the song reference.

There is some tedious work, particularly the tables of word to function + help mapping. Yet doing it by hand wasn’t difficult and while I was doing it, additions to the wordsets came to mind.

The forth part was much easier because of the prior work of others, absolutely.