EFFORTPOST [EFFORTPOST ] Porting TempleOS to user space: a blogpost

TempleOS, that esoteric operating system developed by a schizophrenic guy who loved saying the n word. You've probably heard of it and occasionally get reminded of its existence when you see and . It's probably of no value and only /g/ schizos use it to get called a gigachad, don't they? Who else could _possibly_ use it? Well, here's one that went as far as to porting it to ring 3 and blogposts about it on rDrama of all sites because hes too lazy to even setup a https://github.io blog.

Ok, interested? So basically I effectively made TempleOS an app that can be launched from Linux/Windows/FreeBSD and be used as either an interpreter that could be run from the command line, or as just a vm-esque orthodox TempleOS GUI that you could use just like TempleOS in a VM, just faster (no hardware virtualization overhead) and more integrated with the host. It doesn't have Ring 0 routines like InU8 so it doesn't have that "poking ports and seeing what happens" C64core fun though, so keep that in mind.

It also has a bunch of community software written for TempleOS like CrunkLord420's BlazeItFgt1/2, a DOOM port, and a CP/M emulator written in HolyC. Try them out! There's also networking implemented with C FFI and an IRC server+client and a wiki server in the repository that uses it, if you're concerned whether Terry would think it's orthodox, it's totally ok. You could read more about why in the repo readme.

Here's a simple showcase, this would show you the gist/rationale of making this software.

So let me go on a journey of longposting about how I ported TempleOS to Ring 3.

Step 1. The kernel

There's a _lot_ of stuff in TempleOS that's ring 0 only. No wonder, since Terry always was adamant about being able to easily fuck with the hardware in a modern OS. But on the other hand, this makes porting TempleOS a _lot_ easier. Since the whole operating system is ring-0 only and is a unikernel, every processes share the same address space and that means you could run all the kernel code confined in a single process running on top of another opreating system and have no problems with context switching and system calls, they're all just going to be internal function calls inside the process itself.

Now with this idea, what could we do? We have to study the anatomy of the kernel to be able to port it, of course.

This blog explains it in much more detail, but here's the gist:

You have a bunch of code, but it's incomplete. There's a "patch table" that has the real relocation addresses for the CALL addr instructions, and you fill them in, this sounds easy enough. Plus, TempleOS already has the kernel loader written. We're sneeding that.

Here's some of the code, but it's irrelevant. Let's move on.

But wait, it shouldn't be this easy. How does TempleOS layout its memory? Let's check the TempleOS docs. Did I mention TempleOS is far more well documented than any other open source software in the world?

Step 1.5. The memory and the poop toad in the secret sauce

That's amazing! Let me quote a few important parts:

>64-bit mode requires paging, however, so it is identity-mapped -- virtual identical to physical.

>In TempleOS, the lowest 2Gig of memory is called the $FG,2$code heap$FG$. TempleOS's compiler always uses 32-bit signed relative JMP & CALL insts because 64-bit CALLs take two insts. With signed +/- 32-bit values, code can only call a function within 2Gig distance. Therefore, TempleOS keeps all code in the lowest 2Gig memory addresses including what would normally be called "the kernel". Two Gig is plenty for code, don't worry.

So what does this mean? We need RWX (read+write+execute) memory pages mapped in the process' lower 2 gigabytes, and normal memory maps anywhere else. This is great because we don't have to care too much about where we should place memory. Plus, memory is "identity-mapped" (host memory would directly mirror TempleOS' internal memory addresses) so no worries about address translation.

Here's the code only for the POSIX part of the virtual page allocator because it's more elegant. It's a simple bump allocator with mmap. Works on all Unixes except OpenBSD because they won't allow RWX unless you do weird stuff like placing the binary on a special filesystem with special linker flags because of security theater measures. Theo, nobody uses your garbage.

End step 1.5

We're going to have to strip out all the stuff that does ultra low-level boot/realmode stuff. This was a really long tedious thing to do and I'm not enumerating everything I removed. Ok, so what's next?

Step 2. Getting it to compile

How are we going to get this to compile in the first place? Well, turns out the HolyC compiler can AOT compile too alongside the JIT compilation mode it's known for.

So we write a file that just includes all the stuff to make a kernel binary. This part is very short but we're in for a ride, bear with me.

Step 3. i can haz ABI plz?

How do we call HolyC code from C, and vice versa? Intuitively you should know that this is a requirement. Let's check the docs again.

Ok, cool. This means that

1. TempleOS ABI is very simple

All arguments are passed on the stack, Variadics are also very easy. Take a look at this disassembly:

PUSH 0043BBED pushes the address for the string "Hello rDrama" on the stack, PUSH 17 pushes 0x17(23), the second argument to the stack. PUSH 02 means there are two variadic arguments, and you could access it from the function as argc. argv points the start of the two variadic arguments we pushed on the stack. Much simpler than the varargs mess you see in C, right?

Here's an additional diagram for a C FFI'd function that the HolyC side calls variadically.

2. FS/GS is used for thread-local storage - the current process in use and the current CPU structure the core has.

this is VERY important. Don't miss this if you're actually reading this stuff. All HolyC code is a coroutine, you call Yield() explicitly to switch to the next task. There is no preemptive multitasking/multicore involved. Everything is manual. Fs() points to the current task which gives HolyC code a OOPesque this pointer that you pass in routines involving processes, and you can use them for any process - be it yours or another task and easily play around with them.

So how do we implement Fs and Gs? Simple, thread-local variables in C. We'll come back to C very soon

Sorry if you were disappointed in the implementation, lol

3. Saved registers

You save RBP, RSI, R10-R15. That's the only requirement for calling into/being called from HolyC.

Here's how I implemented HolyC->C FFI. Save all the HolyC registers, and have placeholders for CALL instructions that you fill in later, kinda like the TempleOS kernel itself. We move the HolyC arguments' starting address to the first argument in the host OS' calling convention, so an FFI'd C function looks like this:

How do we implement C->HolyC FFI btw? Well, it's vice versa, but this time we push all the host OS' register/stack arguments on the HolyC stack.

I wrote a complete schizo asm generator for this that I assemble & link into the loader.

Step 4. Seth, bearer of multicores

The core task in TempleOS in called Seth, from a bible reference. This turned out to be relatively easy, after we load the kernel and extract the entry points from it, we simply execute all of them in the core with the FFI stuff we just wrote above.

This shouldn't be this easy. What are the caveats? And what are those signal handlers?

Well, we need to add Ctrl-Alt-C support, which is basically CTRL^C in TempleOS. HolyC, as mentioned above, doesn't have preemption, so an infinite loop without a yield will freeze the whole system. How do we break out of this?

We use signal handlers in Unix for this. Basically we use the idea that the operating system will force execution to jump to the signal handler when a signal is raised.

On Windows, it's a bit more sassy. Windows has the ability to suspend threads remotely and get a dump of the registers, and resume it.

Step 5. User Input/Output

I use SDL for the GUI input/output and sound (TempleOS has BIOS PC speaker mono beeps for sound, it's very simple), and libisocline for CLI input. I'm not going into super specifics because it's boring as fricc.

Step 6. Filesystem integration

TempleOS uses drive letters like C: and D:. We need to translate these ondemand for the kernel to access files.

This is the heart of the virtual filesystem. It's quite simple. We just strcpy a directory name into a thread-local variable, and basically have an alphabet radix table.

I just wanted to show you guys this part. It's a file truncation routine & its super lit, we can throw HolyC exceptions from C because throw() is a function in HolyC.

Small "logic switch" thing I did for the poor man's Rust match, thought it was neat. (Writing EXODUS in Rust would not be fun. Unsafe Rust gets ugly quick, and I've tried writing some unlike the /g/ chuds)

Step 7. Debugging

Now, we've almost reached the end. At this point, you can run stuff just fine with our TempleOS port. But, how do we debug HolyC code?

TempleOS has a very, _very_ primitive debugger. I thought this was _too_ primitive for my taste, so I gave it a modern spin:

Looks much better, and more orthodox in a way.

I just dump all the registers when I catch a SIGSEGV or anything else that indicates a bug and send it to the HolyC side.

Step 7.5. Backtrace

How do we get the backtrace of the HolyC functions? Fear not, because the kernel calls a routine that adds all the HolyC symbols to the C side's hash table in the boot step. Now that we have all the symbols what do we do?

Here's the anatomy of an x86_64 function if you don't know:

RBP(stack BASE pointer) points to the previous RSP(stack pointer) of the callee of the function, and RBP+8 points to the return address, which means where the function, after returning, will resume its execution at. Now with this knowledge, how do we implement a backtrace?

We keep drilling down the call stack and grabbing RBP+8's so we know which functions called the problematic function, and find the address offsets in the symbol table with a linear search.

end step 7.5

Congratulations, this is the end. This probably covers more than your average university CS semester. My stupid ass can't articulate this in a juicier way sorry.

Trivia

• Terry never used dynamic arrays (vectors). He always used circular doubly-linked lists because they're much more elegant to use in C. Really, there's no realloc too. (<(data locality be damned) its actually not that bad.)

• HolyC typecasts are postfix, this is to enable stuff like HashFind(...)(CHashSrcSym*)->member which makes pseudo-OOP much cleaner. HolyC has primitive data inheritance. (this one is code to retrieve a symbol from a hash table, HashFind returns a CHash* but CHashSrcSym inherits from CHash)

• "abcd %d\n",2; is shorthand for printf

• HolyC has "subswitches", like destructors and constructors for a range of cases.

Cool, huh? It's very useful for parsers.

• I mentioned this eariler but let me reiterate: All HolyC code is a coroutine. You explicitly yield to Seth, the scheduler, for context switching between tasks. This is how cooperative multitasking should be done, but only Go does it properly, but even then they're not the real deal by mixing threads with coroutines.

• IsBadReadPtr() on Windows friccin sucks. Use VirtualQuery. You can do the same thing in Unixes with msync(2) (yeah wtf. it's originally for flushing mmapped files but hey, it works)

• There's a ton I left out for the sake of brevity but I invite you to read the codebase if you want to dig deeper.

Big thanks to nrootconauto who introduced me and led me through a lot of Terry's code and helped me with some HolyC quirks. He has his own website that's hosted on TempleOS and it's lit. Check it out.

There's probably more but I think this is enough. Thanks for reading this, make sure to leave a star on my repo if you can