Local Inference with Large Language Models

To head off any criticism: not a word of this text was written by AI, all the mistakes are my own. If you should spot any I’d be very happy if you let me know.

This article assumes basic familiarity with the terminology around LLMs and the various moving parts, I’ve tried to explain as much as possible but in case something is unclear feel free to reach out and I’ll expand the article or clarify in an email.

Why?

Why make a setup to run an LLM locally if the bleeding edge is affordable and easily accessible?

I’ve been playing with LLMs since the first open models were released and even though at the time the results weren’t all that impressive it was clear that it wasn’t going to stop there. Over the last couple of years the model capabilities have improved at a breakneck pace and today these models are extremely capable when used appropriately. The business that I run sells a service to investors and companies that are looking to be acquired and is highly confidential in nature. Effectively we ask other companies to share pretty much everything, both the good and the bad with our team so that we can come to a balanced verdict in our report. To enable this sharing we sign NDAs (non-disclosure agreements) that usually have scary numbers in them in case we would breach that confidentiality, and rightly so, this kind of data is easily weaponized.

As part of our process we write reports, our ‘work product’ and these reports need to be proofread. The problem is that typically these reports are due on Friday evening and our draft version is usually only ready somewhere between 2 and 3pm on Friday afternoon. That leaves very little time for proofreading, usually we try to get one or two passes in before the document is handed to a customer but it isn’t rare at all that on second or third reading during the weekend we still find mistakes. These can be small and innocent or they could be larger (for instance: typos and grammar errors or errors in the content itself). Given infinite time I’m sure we would find all of these but we simply don’t have that luxury.

Now, the easy way out would be to upload the document just prior to sending it out to an AI service provider, have that service do the proofreading and then we would apply the corrections (there is absolutely no way that I would allow an AI control or direct input into our work process). But that would cause us to break our NDA. And that’s before we get into the hairy details of questions such as ‘Can we trust Altman or Amodei when they say they won’t use the data for their training?’, ‘Even if we can trust them, can we trust everybody who works there?’ and ‘And even if we can, are we sure that shipping these files out at all is a wise decision?’. The answer to the last one was the easiest, that was simply ‘No.’. So we decided to bite the bullet and do this locally.

We have some other reasons as well: we don’t particularly like the idea of a large entity vacuuming up the whole of human culture and then selling it back to us, no matter how nicely packaged and how useful it is. That feels like a very unethical thing to do and the providers of ‘open weights’ models are closer to how we think this should be handled. Second to that is that we don’t want to end up in a situation where an outside party has control over our workflow and can tell us what we are - or are not - allowed to do. ‘Sorry Jacques, I can’t let you do that’ is not something that I would ever accept from a computer, computers do what I tell them to do, not what they will allow.

These locally run models are not precisely as good as the big frontier models. They tend to be a bit smaller, sometimes a bit older and probably not quite as clever. But depending on your workload and other demands (such as confidentiality) they can be a surprisingly good match. And in case the likes of OpenAI decide to hack your server they may well be your only means of defense.

What do you need to do local inference?

If you want to do inference locally you’re going to need a couple of things:

• A piece of hardware powerful enough to do this on
• A model that will fit that hardware
• An inference engine that will push your input through the model to generate the required output

Let’s start with the last one.

Inference Engines

There are a small number of major engines suitable for deployment on your own hardware, a much larger number of derivatives (sometimes weakly derived, sometimes not much more than a relabeling) and then there are numerous smaller efforts. We only considered open source engines, and more specifically engines that we have actually used (benchmarked, and used in production). Let’s focus on the main engines first. These are the ones that we used that are either the ‘root’ projects or independent projects that have shown sufficient staying power that you could presumably build a business on them:

Llama.cpp

Llama.cpp (WP) is probably one of the easiest ways to get started with local LLMs. The project is now 3 ¹⁄₂ years old, stable and extremely versatile. Installing it on a wide variety of hardware is usually a matter of cloning the GitHub repository and a single command to configure and another to build it. It’s not necessarily the fastest inference engine but it is super reliable and the quality of the code is quite high. Developed initially by Georgi Gerganov who now manages the project together with a team of seasoned developers. Georgi still contributes quite a bit to the code. We have used llama.cpp on pretty much all of our hardware and with a really large variety of models. Occasionally you’ll run into a lack of support for a model that has just been released but usually within weeks (sometimes days) these things are resolved. Llama.cpp is not usually the inference engine used by commercial entities, where the last little bit of performance weighs heavier than the convenience factor that Llama.cpp brings to the table. The project is named after the ‘Meta’ (ok, Facebook) model.

Good guides exist on how to get it working, the file format (.gguf) is widespread and conversion utilities are available in case a model is not shipped in a format that Llama.cpp can use. There are also groups specializing in converting and quantizing models to make them fit on particular hardware. Updates are a daily affair, it is usually a good idea to keep an eye on the GitHub page to see what they’re up to and whether or not it may be useful to you.

vLLM

VLLM is a powerful inference engine that is far more complex than most other engines. The downside of all this complexity is that installing, configuring and running it can be a significant task. The upside is that it tends to extract more performance from the same hardware than other engines, if that hardware is a good match for VLLM. If you intend to use VLLM you need to ensure that the hardware is a good match, usually by purchasing the hardware with the specific goal of running VLLM. Manufacturers of AI specific hardware may choose to support the ‘recipe’ for their hardware to ensure that VLLM runs well. This can be a shortcut to a powerful solution. But figuring this all out on your own can be extremely time consuming. VLLM is the inference engine of choice for many commercial LLM service providers. For them it is simply a matter of money, it is cheaper to invest the up-front effort and then to reap the benefits from a (usually marginal, but there are exceptions) system that performs better.

vLLM setups tend to be a bit more fragile, require more expertise and are less likely to function well ‘out of the box’. But if you are willing to spend the time, and already have some expertise this may well be a good match for you. VLLM configurations typically use the safetensor format that most models ship in from the model provider, so normally speaking no conversion step is necessary.

SGLang

SGLang sits somewhere in the middle between vLLM and Llama.cpp when it comes to ease of installation and ease of use. It usually squeezes a bit more performance out of the hardware than Llama.cpp but this comes at a price, we found that for our workloads on a pretty varied bunch of GPUs we did not see enough gain in performance to warrant further pursuit, but it is possible that we simply did not spend enough time to really unlock this engine. Installing it is a matter of running a docker container. This makes auditing what you are running a bit harder than with the other two packages (an area where Llama.cpp wins hands down because of its simplicity).

DwarfStar

DwarfStar is an inference engine produced by Salvatore Sanfilippo of ‘Redis’ fame. Salvatore is an extremely accomplished and productive programmer and DwarfStar is an interesting variation on the inference theme. Rather than make an inference engine for a hundred different models and 500 different hardware combinations he focused on (initially) a single model (specifically DeepSeek, aka DS, which is probably where the project got its name) and a Mac with unified memory. Since the launch many more platforms are supported as well as several more models. DwarfStar is easy to use and easy to configure, it is regularly updated with new features and/or performance improvements.

Ninfer

Ninfer is, like DwarfStar, a specialized engine. Initially made to run Qwen 3.5 and since adapted to be able to run other models as well it squeezes every last drop of performance out of a single NVidia 5000 series GPU. If your setup matches that description then this is most likely the best engine for you assuming your model is available as a ‘.ninfer’ artifact (beware, there are multiple versions of these that are not backwards compatible so you’ll have to convert). Ninfer is in active maintenance and, interestingly, there are already many forks for hardware that is similar but not identical to the hardware that ninfer was originally developed on. For instance, there are forks for the 3000 series GPUs (which includes the venerable 3090, which is still a powerhouse, especially given its age) and the 4000 series as well.

Does it matter which engine you choose?

Well, it does and it doesn’t. As long as the engine exposes an OpenAI compatible endpoint you don’t necessarily care all that much about which engine you are running, as long as (1) it runs well, (2) performance is acceptable and (3) your model of choice is supported.

The Models

There are thousands (if not tens of thousands) of different models. These range from general purpose models targeted to specific sizes of VRAM or some other inference time limitation to models tuned for a particular application. The main factors distinguishing the models from each other are:

• The number of parameters (anywhere from 1B to > 1T)
• Whether the model is ‘dense’ or a ‘Mixture of Experts’ (‘MOE’)
• Whether the model has a built in chaperone or not (the term to look for is ‘Abliterated’)

We could not possibly list them all, but at the time of this writing there are a couple of these that deserve special mention here:

The ‘Qwen’ family of models

The Qwen family of models is interesting because they span a range of capabilities and sizes, from small (7 GB or so, small enough for even the smaller video cards) all the way to 400GB and by virtue of quantization many different sizes in between. One of our workhorse models is Qwen 3.8 27B, we use this model for coding and robotics experiments (not the original application of our AI rig, but now that we have it, why not…).

Deep Seek

DeepSeek is a very impressive model, in fact, it was the first release of DeepSeek that prompted us to make the next step with this project because we considered the quality ‘good enough’ for our use case. It is not necessarily the best model overall but it strikes a good balance between speed and size and is quite versatile.

GLM

GLM, in particular GLM 5.3, is an extremely capable model. It is also the largest model that we can still comfortably run on our hardware. Due to its size it is not super fast but it more than makes up for that in capability, so much so that it can perform better in wall-clock than smaller models that will use many more tokens and/or many more turns to get the same job done. GLM 5.3 is also very good at cyber related tasks. To this day it has never refused an assignment.

This is the model that we use right now for our in-house proofreading project and we have started using it for other purposes as well.

Does it matter which model you choose?

Absolutely, the choice of model is probably the most important choice you can make when you decide to do local inference. If at all possible test-drive all of the models you are considering, and don’t limit yourself to the ones above, these are simply the ones that we use most frequently but there are many, many others besides that may be more suitable for your purpose. If you are knowledgeable about this stuff you have probably already noticed that all of the models mentioned above are Chinese. This is no accident: China is after the United States the AI powerhouse and because they can’t or don’t want to compete directly with the big US service providers they’ve decided to open up their models. Yes, that’s right, commercial entities are sharing their work product for free with the world. Predictably, the response from across the Atlantic (or the Pacific, depending on your point of view) has not been kind and there has been an endless barrage of fearmongering, attempts at regulatory capture and other shenanigans in an to attempt to lock in the market for the United States. We believe that when being given the choice between running a ‘top of the line’ model but giving our data to parties that will no doubt use it against us and running things locally, we believe we’re doing the right thing by picking the latter. If Chinese models are not an option for you then there are still many options left. But the Chinese models are hard to beat both in price (you really can’t beat free) and in performance, as long as you steer clear of sensitive political content they should be fine.

Hardware

So, now for the meat of the matter. The inference hardware. This is where things get complicated and expensive in a hurry. Well, what’s expensive. If you had offered me a black box with a power cable, an ethernet connection, 3 KW of power consumption and the capabilities of any of the models mentioned above you could have named your price and I would have bought it. The business angle is very obvious and we (my colleagues and I) are running a business, so the decision to spend some money on these capabilities is an absolute no-brainer. At the same time: what do you get for all that money?

Let’s look at a couple of use cases:

• hobby / home use

• small business (say single proprietor, maybe a legal practice with one or two lawyers)

• larger business (5-500 employees, one or more teams)

Note that for all of these scenarios from an economic perspective it is cheaper to use a hosted service provider, but I trust you have your own very specific reasons for why you want to do this under your own banner.

Hobby or home use

If you are looking at doing this for hobby reasons, to learn more about LLMs and/or to play around with them in different application settings fully under your own control, or maybe to develop some software that closely integrates with the LLM (for instance, a custom agent) then you are probably going to be using a single GPU and a ‘fat’ PC with as much RAM as you can stuff in it. 32GB and a 16GB VRAM GPU are probably the lower limit of what is usable, anything bigger than that and you will have more speed and/or a more capable model. With a nice gaming PC (and paired with an equally nice GPU) you could have a very capable setup well under $10k, assuming you are buying new or refurbished hardware.

Small business

For a small business the situation is not all that different from the hobbyist situation. You probably will have an easier time justifying the investment and you may have an actual business reason to do so rather than just curiosity or the ability to develop software closely tied to the LLM itself. More likely you will be integrating the LLM ‘as is’ into your workflow to work with documents, help with writing and/or to assist with QA. That fat PC and GPU mentioned above will still be a good starting point, but given that this is a business expense you will likely have more money available than the typical hobbyist and you’ll be able to afford more RAM and more VRAM so you’ll be able to run larger models. $10k is again roughly the entry level, you can go up as far as your budget and ambition allows.

Larger business

This is where it gets complicated. If you already have IT staff then configuring and maintaining an in-house LLM setup is probably best farmed off to the people that already know how your business operates from an IT point of view. A large dutch Legal services firm did just that and is spending a substantial amount of money on their internal AI project. Effectively the sky is the limit with such projects. But depending on how large your operation is you should expect to spend between 5 and 10K per person that uses the system, and as the setup grows you will begin to see some economies of scale.

GPUs

We only have experience with one brand of GPU, NVidia. In our own lab we have various GPUs, 3090’s (6x), 4000’s (6x) and some odd ducks, notably two DGX Spark AI mini pc’s. All of these were bought second hand with the exception of two of the 4000’s and the DGX Sparks.

RTX 3090

The 3090 is - in spite of its age - still a very powerful GPU. It’s a bit power hungry relative to the performance but the memory bandwidth is very impressive and there are peer-to-peer and other performance hacks floating around that will get you a large amount of bang for your buck. The second hand prices of these (you can no longer buy them new) are as high or higher than the original new price, which is something that does not normally happen with computer hardware. There are many different versions of this card, some with power limits as high as 450 Watts and with various kinds of cooling systems (water cooled, air cooled, with a ‘blower’ style fan or with regular fans). I personally prefer the Asus ‘Turbo’ blower style version, it is only two slots wide instead of three and it seems to stay cooler than the others at the same ambient temperature. I power limit the RTX 3090 to 275 Watts, this gives a bit less performance but far less heat and I hope that extends the life of the cards. All of these have 24 GB of VRAM.

RTX PRO 4000 (Blackwell)

There are multiple RTX PRO 4000 series GPUs so beware you’re not buying the older ones (unless you are looking for those specifically). The RTX 4000 Blackwell is a very handy GPU because it is a single slot wide. This allows you to either cram in a lot of them in relatively little space (beware of cooling issues!) or you can space them out so they have a bit more air to breathe (and the slots in between can still be used). These cards sip power compared to the older 3090’s, offer roughly equivalent performance (in spite of having much lower memory bandwidth) and are still available new (at a price though, roughly 2x what you pay for a used 3090).

RTX 4090 (Ada Lovelace)

The RTX 4090 is a lot more powerful than either the RTX 4000 or the RTX 3090, even though it is older than the RTX 4000 Blackwell series. The key is the generation difference with respect to the 3090 and a massive power budget compared to the 4000. You can expect ~50% more speed from these cards. An interesting twist to the whole 4090 saga is that Chinese re-manufacturers have figured out how to double the VRAM on these cards, so you can buy ‘new’ cards (with a used GPU and probably used VRAM but a new board and cooling system) that have 48GB of VRAM. The originals have 24GB. Note that these souped up cards are not sanctioned by NVidia and that there are issues with the BAR1 window being smaller than the available amount of VRAM and not all engines are smart enough to compensate for that, especially not when using peer-to-peer transfers between cards.

Other GPUs

There are many other GPUs, but the ones above are the ones that we have and that we have used extensively. We haven’t had a single GPU die yet so they’re pretty tough (in spite of running them pretty much day and night under varying loads for a long time). Other interesting options are the RTX 5090 (which right now is approximately worth its weight in gold), the RTX PRO 6000 Blackwell MaxQ (which has 96 GB of VRAM, but costs as much as a mid sized car) and some more exotic ones such as the V100s (older but still very capable), the GB200 and the GB300 (high density data center GPUs, extremely powerful, extremely power hungry, will probably require babysitting and a server room). Then there are AMD which make some very capable GPUs, Intel, IBM and more specialized hardware, but which you are unlikely to encounter in the wild.

DGX Spark

The DGX Spark is an interesting development. We have two of these and they’re pretty capable little machines. Each Spark has 128G of unified RAM, a Blackwell series GPU and an ARM CPU. All of this is squeezed together in a tiny little box. They’re getting more expensive by the day (last I checked about $8K, we got ours a lot cheaper than that). They’re not the fastest because even though on paper those specs look absolutely amazing they have only ~275 GB/s memory bandwidth, which is 1/3th of what that much older 3090 brings to the table. So performance is lacking. But they are quite convenient and they will run relatively large models. You can gang them together using a high speed interconnect. This is useful but the experience is not as seamless as with multiple GPUs connected to one CPU and besides that it isn’t really much faster than a single Spark. But it does get you the capability of running larger models with a minimum of fuss.

The host

Hosts you can configure pretty much any way you want, but there are a number of things you will want to watch out for: try keeping the amount of RAM that you have at least as large as VRAM and largest model size with some room to spare. For hobby use 32 or 64G, for professional use 128G and up (if you can afford it…). This will really help with loading models quickly and it will also help you shuttle bits of the model in and out of the GPU(s) at speed. A fat CPU helps as well, we use an AMD Threadripper PRO 5955 which has plenty of PCIe lanes (this can be a serious showstopper on some intel based hardware) and a motherboard that has seven slots. Make sure your BAR1 resizable setting in the bios is enabled and that you allow for > 4GB decoding on the PCIe bus.

Typically motherboards will allow up to 14 GPUs before they start refusing to enumerate more of them. This, and the physical size of your installation (our original ‘Frankenrig’ had that many GPUs and it was absolutely unmanageable) are things to consider and it may well be worth it to get fewer, larger GPUs than many more smaller ones. Some people have managed to get 16 GPUs to work but they did not give any pointers on how to achieve this.

You also want the fastest and largest NVMe drive that you can afford. This will help tremendously with model start-up time and if you are going to run even larger models than what fits in RAM you can use that capacity to stream the model weights in as required. I would not recommend such a setup for professional use though.

More GPUs than you can fit in the box

This is where things get tricky. At some point your desire for VRAM and/or Compute capabilities my outstrip the space in a single enclosure. This is a huge bottleneck and many people stop right at that point. There are a couple of solutions to this problem and I’ll sketch them both below.

Risers and splitters

PCIe, the bus that is used to tie all these parts together is incredibly versatile. What you think of as a way to plonk expansion cards into a motherboard is actually an extension of the CPU that is more like a local network of parallel connections than that it is just a point-to-point link between the CPU and a card. The practical upshot of this is that you can add more than one card in the same slot. Ok, you won’t be able to do so physically, after all the GPU exclusion principle prohibits two cards from occupying the same space. But what you can do on some motherboards is to tell the board to ‘split the lanes’. A full sized slot is x16, so 16 parallel lanes. You can hook up two cards, one using x8 to one device and the other x8 to the other device. Both will run at correspondingly reduced bandwidth but if your original motherboard had 7 slots using this trick you can now run 14 GPUs (this was our original Frankenrig). Mechanically it is going to be a bit messy. If you want full speed another alternative is to use slim (single slot) cards like those RTX PRO 4000’s mentioned above, and then to put riser cables in the slots in between. These riser cables can be x16 and then you can stuff 4 slim cards and 3 full sized cards into the same box at full speed. Again, mechanically this is a bit of a hassle, but it can be done. If you’re in Europe there a little company called C-Payne that sells quality hardware for this purpose, we’ve bought plenty of their gear and all of it is top notch.

Switches

Switches are an absolutely brilliant way of expanding the PCIe bus from a computer. There are a couple of advantages, this setup will be far more stable mechanically than anything involving risers or splitters. In theory you can expand this as much as you want (BIOS enumeration limits still apply though, let me know if you know a way around this), and it also solves another problem, which is to power that many GPUs responsibly.

This is the road that we took with our current installtion. We have a big CPU box with 4 GPUs in it, then an expander card that connects to the switch board. The switch board has its own PSU and hosts the second set of 4 GPUs. Besides the power problem this setup is also much, much easier to keep cool than to have that many GPUs in a single housing.

The switch board we use is made by Chinese company ADT. The quality of the board is very high and it works out of the box with Linux, no drivers are required. ADT sells all kinds of other goodies as well for PCIe wizardry, but this is the only component we use from them at the moment.

There are other boards in this category, they sell a 5 slot board and there is also ‘Utran’, who sell an even larger one. These boards are not cheap, about $1000 give or take and you’ll need to build an enclosure for them because the spacing between the cards is non-standard. I made a quick-and-dirty frame out of some 2020 aluminum that I bought in Poland, putting it together is something anybody that can build a Lego set can do. To hold the cards, the powersupply, the fans and to mount the board to the box I 3D printed a bunch of parts. If you want copies let me know I can have more of them printed easily enough (probably not worth the shipping if you are in the USA).

Putting it all together:

The Frame

PSU mounted

Switch board mounted

48 GB Remanufactured ‘white label’ RTX 4090

Four GPUs on the switchboard:

Two PCIe expanders next to the host computer:

And when you’ve done all that you will see something like this:

  |   0  NVIDIA RTX PRO 4000 Blac...    Off |   00000000:01:00.0 Off | Off |
  |   1  NVIDIA RTX PRO 4000 Blac...    Off |   00000000:2C:00.0 Off | Off |
  |   2  NVIDIA RTX PRO 4000 Blac...    Off |   00000000:41:00.0 Off | Off |
  |   3  NVIDIA RTX PRO 4000 Blac...    Off |   00000000:61:00.0  On | Off |
  |   4  NVIDIA GeForce RTX 4090        Off |   00000000:66:00.0 Off | Off |
  |   5  NVIDIA GeForce RTX 4090        Off |   00000000:69:00.0 Off | Off |
  |   6  NVIDIA GeForce RTX 3090        Off |   00000000:6D:00.0 Off | N/A |
  |   7  NVIDIA GeForce RTX 3090        Off |   00000000:6E:00.0 Off | N/A |

Epilogue

What does it cost?

The total cost for the system in the picture above is ~$25K. That includes 256GB of RAM, 8 GPUS with a total of 240GB of VRAM, two very fat power supplies (1600W each), the Ryzen 5955 CPU, a motherboard and all of the cables, mounting hardware etc. Not cheap, I’ll admit. But then again: in 1995 I paid a similar amount of money for a decked out workstation and once upon a time I bought a laptop that was half this price. Computing hardware is - and probably always will be - expensive. I doubt you could build this system for the same amount of money today, the prices have gone up considerably for the GPUs, to the point that some of them are now effectively unobtanium. The four RTX PRO 4000’s alone would run to half of the total today.

What does it do?

Running 8 concurrent Qwen 3.8’s using ‘ninfer’ each of which is running 8 slots you can get a few thousand tokens generated per second. Prefill is ridiculously fast, 2K tokens / second, time to first token on shorter prompts is instantaneous (too short to measure, really) and on longer prompts (such as a long document to proofread) mayb 20 seconds or so.

GLM 5.3 is slower, much slower. By itself it occupies all GPUS. There is a little bit of room to spare but not all that much:

  |    0   N/A  N/A           26087      C   build/bin/llama-server                13456MiB |
  |    1   N/A  N/A           26087      C   build/bin/llama-server                22700MiB |
  |    2   N/A  N/A           26087      C   build/bin/llama-server                22700MiB |
  |    3   N/A  N/A           26087      C   build/bin/llama-server                19234MiB |
  |    4   N/A  N/A           26087      C   build/bin/llama-server                41456MiB |
  |    5   N/A  N/A           26087      C   build/bin/llama-server                37328MiB |
  |    6   N/A  N/A           26087      C   build/bin/llama-server                22656MiB |
  |    7   N/A  N/A           26087      C   build/bin/llama-server                24044MiB |

The number of tokens generated per second is anywhere between 50 and 100 depending on how well the speculative decoder does it’s job. Unfortunately llama.cpp does not really get the maximum out of this setup yet, GPU utilization is fairly low while it is running. Prefill is reasonably fast at 500 tokens/second.

Power consumption is obviously nothing to be proud of. Running flat out this rig would consume well over 3 KW. I’ve power limited the 4090’s and the 3090’s so things are a bit more civil now but still, it’s like having a very fancy space heater. There are a lot of solar panels on this building so that helps considerably (but only during the day).

GPU SCAMMER Alert

A friend of mine, an immigrant to the Netherlands just got scammed. Someone offered a GPU for sale and he committed to buying it. They agreed to meet and money changed hands, the guy jumped on an e-bike and ran off. It turned out that the GPU and the VRAM had been removed from the card making it worthless, leaving my friend with a 3000+ euro problem. He’s a student in the AI/Robotics field so for him this is a very large amount of money to lose.

He went to the police to file a report but it turned out that all of the data the scammer had provided was fake. My friend did have one ace up his sleeve, he had made a picture of the person that was dropping off the card. So if you come across this character: Do not do business with them. The platform this happened on was the Dutch website marktplaats, a local marketplace that is owned by Ebay. If you have any data on this scammer I’d be more than happy to forward it to my friend and to the police. The picture was taken near Zwolle train station.

Now, obviously this is very annoying for my friend, but the main reason I’m posting this is that I want to warn anybody that is shopping around for GPUs right now:

The market is crazy. There are plenty of scams, and there are people unloading stolen property. Treat any kind of deal with great care, don’t deal in cash, require to see identification and only deal with people whose house address you’ve verified. It’s a complete wild west and it will continue that way until prices normalize again, which could be a while.

The Problem

There is a problem in the ‘infosphere’. In a nutshell: we now have high quality text that has low, zero or even negative value for the reader. Authorship is no longer something you can trust implicitly, even science is not immune to this and the problems seems to be set to become significantly larger in the near future. You could solve this by mandating authors label their work product but if there is money to be made by mis-labeling then you can expect people to do so.

Wikipedia has been fighting this trend for many years, they position themselves as a tertiary source: they will require references from secondary sources and those secondary sources should report on primary sources. This serves as an input filter of sorts because the assumption that secondary sources are going to do their homework was a good one. Until now.

Secondary sources are now incentivized to no longer do their homework. Just about every secondary source now uses automatic ways to generate content where it seems quantity, rather than quality has become a driving force. This is the equivalent of replacing quality home cooked food with fastfood: it is much easier to produce low quality food in quantity than it is to produce high quality food in quantity, if it can be done at all.

AI is a force multiplier here: it allows for the generation of immense quantities of very good looking text but with low quality. The information diet equivalent of fast food. And unlike fastfood, which usually is at least nourishing even if it isn’t the best of foods, the AI work product as a rule is poisonous: it contains ‘hallucinations’, which when allowed to take root can cause false information to spread widely. Now, it could be argued that fast food is when taken in quantity also poisonous so maybe the analogy is stronger than I intend for it to be. But the fact is that a single hamburger is not going to kill you. But a single falsehood just might, especially when the subjects are health, technology, engineering and so on.

Of course ‘unassisted humans’ also make mistakes. But we have ways to deal with that: reviews and editors. At the quantity of information produced that is manageable. But with the volume of automatically generated text exploding and absolutely dwarfing our combined work product it is a matter of time before the ‘slop’ will outnumber the combined human product by a considerable margin. Humans won’t be able to review or edit any of that simply because there wouldn’t be enough humans to do and besides the whole point seems to be to not employ all of these humans in the first place.

A possible solution

There are a couple of solutions but they are all going to rely to some degree on collaboration, ethics and honesty, both corporate and individually. If the advertising industry has taught us something then I guess that we do not have to hope for this to be successful. And bad actors will be able to put one over on good actors by gaining an unfair advantage. And since we only seem to want to punish people who gain an unfair advantage in life by clobbering old ladies over the head to make off with their handbags I have little hope that this will succeed. But I still think it is worth trying because the idea of giving up what we’ve achieved so far seems to me to be too high a price to pay.

AI - obviously - has its place. It can help us to have a personal teacher - who you can’t quite trust not to pull the wool over your eyes - with access to a vast amount of information. But as soon as you hit the ‘prompt / generate / cut / paste / profit’ cycle you are no longer operating from the same playbook as the rest of humanity: you are passing off the AIs work product as your own, whereas all you did was to create the prompt. And that’s fine, but then you should at least be transparent about it.

In order to help AI companies to create better products it would be good if we could create the digital equivalent of a sworn statement of authorship. I can see several ways of doing this and one in particular has my fancy because (1) it scales and (2) it would be very easy to implement.

Classification

Let’s start off by grading the different kinds of works

• Original works. These are 100% human created, there is absolutely no question about the provenance of the text. We know who the author is, the author has made a clear statement of provenance and there is no reason to doubt that statement.

• Assisted works. These are hybrids: the author has put considerable work into the text but has used AI to either stylistically improve the work or because some of the text has been AI generated. Ideally that would be a minority, as soon as you cross the 50% mark you will be in the next class.

• AI generated, human augmented. These are hybrids too, but from the other side. Now the balance has shifted to majority AI input and minority human.

• AI generated. 100% AI generated, no human input required.

One way to be able to distinguish what is human written and what is AI written would be by extending the Unicode character set with a complete duplicate and to legally mandate that all AI companies are only allowed to use this character set for their output. Transliteration to the human set would be strictly forbidden and should result in some civil penalty. Shipping an AI model that does not output in this transliterated set would result in a criminal penalty.

That’s harsh, yes. But I think that there is enough at stake here that this kind of measure is more than warranted. And AI companies have a huge incentive to play ball here: for one their execs would likely not want to be found engaging in criminal acts, for another they would be able to avoid that one thing that they’re all struggling with: to avoid eating their own (and each other’s) tail. Because AIs trained on slop will eventually degrade to the point where progress will stall or even reverse. Pre-AI text will have roughly the same value as low radiation steel (https://en.wikipedia.org/wiki/Low-background_steel). By labeling their output in this way AI companies will be able to sift out their own and other AI companies’ output.

Finally, a human run attestation service could be run: a sufficiently secure cryptographic hash run on a piece of input text that measures the ratio of AI vs human generated text in the input could give a meta label that can easily be affixed to the content. Search engines and AI ingestion could then use these marks to differentiate between various sources and hopefully use this as a degree of measuring quality and giving higher quality results priority. So you don’t accidentally end up biting into fast food when you were expecting something much better.

Problems with the above

Of course, it is easy to dismiss this all out of hand because it isn’t - and won’t be - perfect. I can already see many sub-problems that will need to be resolve and the knee-jerk reaction will - of course - be that the race is already run and it is much too late to do something about it now so we may as well lie down and be run over by the bulldozer.

But I’m an optimist by nature: I think we can recognize that even if this plan is flawed the spirit of it has merit, fix the flaws and then implement them to the best of our ability. It’s the spirit that counts here, not the letter, so let your imagination loose on taking the idea and improving on it rather than to bash it with the first middlebrow dismissal that comes to mind.

–

This text was written by hand, using a brain and a computer in the most passive way possible, to record keystrokes. No automatic processes contributed to the content. AI percentage: 0.

I’m impressed

For the last couple of years, my colleagues and myself have been playing around with AI. I can’t speak for them but from the perspective of a casual user I am very much impressed by the capabilities on display. If you dedicate some time to it you can get a lot of mileage out of AI, it can help you in the way a highly interactive book would help, with vast amounts of knowledge at the tips of your fingers. As long as you are wary of the inevitable hallucinations and the AI’s tendency to get stuck in loops. Because the one thing that we have not yet managed to program in there is the ability to be humble: to realize that no matter how much knowledge you’ve got you don’t know everything. So “I don’t know” is not on the menu. It is not the kind of thing I have seen as output from the various AI offerings.

If you had told me ten years ago that we would be having conversations with a machine at this level today I would have thought it to be impossible, and yet, here we are. The ‘Attention is all you need paper’ combined with Hinton’s persistence paid off immensely, probably more than even the people behind that effort had expected. Current generative AI is able to converse on an immense variety of topics at a reasonably high level.

Professional use

Even so, I would never use its output in a professional setting. When we are hired to do work we are hired to do work, not to come up with prompts and then to let the AI do the work. Anybody can do that, it would feel like fraud to use AI as some kind of shortcut, then we may as well give our customers the prompts.

As a search engine and uniform interface to documentation though, it is unparalleled, as long as you check the output for factual correctness.

Commercializing AI

Because we like playing around with new tech we put together a monster computer with a lot of GPUs and a very large amount of memory. This gives us the capability to run these models in-house and to evaluate their abilities without accidentally leaking our data to the AI providers. What struck me is that in any conversation longer than a couple of lines you end up with one or more times where you have to correct the AI on some point, after which the conversation may stay on the rails a bit longer. But there is always a point where the growing context becomes problematic and of course when you start a new session all of those corrections are forgotten. That plus any kind of formatting instructions regarding the responses have to be repeated over and over again at the beginning of every new session, a kind of ‘system prompt’ at the user level. Running a system like this commercially requires a lot of power, memory, computation and storage for every a single user. I’m fortunate in that I have a lot of solar panels but if not for that my power bill would be astronomical.

And that’s where the data centers and the price of RAM and GPUs comes in. The price of a stick of DDR5 memory must have the smell of onions baked in because when I look I start crying. This will force the AI companies to commercialize it but their avenues to do so are limited, and I’m not sure how they think they are going to survive without doing some kind of rug-pull.

Confidence

What bugs me about the interaction with the various AI models is that they are so incredibly over confident. They are jump-to-conclusion machines that will answer a one line prompt with six pages of plausible nonsense as often as they get it right. The signal-to-noise ratio can be very low and if you’re not careful you will end up reading a lot of self congratulationary bullshit that does not move the needle at all in terms of your understanding or insight. The number of times I’ve seen the word ‘working’ for something that could not work and ‘no handwaving’ followed by a lot of handwaving is pretty high, at least as high as the number of times that I came away with the idea that I had genuinely learned something. It would be great if the current crop of AI used font size or levels of gray to denote the things they are actually confident about and the things that are speculative. This alone would significantly increase its value to me.

When it does not work

The most use I seem to get out of it is when I’m studying some subject that I already know the basics of to increase that knowledge and give hints on fruitful avenues of reading or research. It rarely happens that the interaction with the AI itself gives direct insight on things that are non-obvious. Over the course of the year that has happened probably a handful of times. The more clueless I am about a subject the less use I get out of AI. Conversely, the more I know about something the more I’m irritated at the basic stuff that it - repeatedly - gets wrong and you end up with endless corrections. It is able to pull a massive amount of information up at a moments notice but it will happily forget something that came up only a few minutes ago. This can be quite annoying, but then, whenever I’m annoyed I am also impressed: I am annoyed at a machine for not being able to so something as if it were a person. And it clearly is not, despite all of the language hints that it is.

The Sleaze

Some of the models that are in widespread use are extremely slick at giving the user the impression that they are smart. They will endlessly congratulate and reinforce that the user is a smart person even when they’re clearly not. The paternalizing attitude is so ingrained that even after multiple reminders it will often just find new ways to effectively do the same thing.

The Hive Mind

After playing with something for a while I always wonder: What’s next? Where are we headed? I think the current crop of AI products is useful, but the push towards commercialization will most likely destroy a lot of that utility. The same has happened with all other media that preceded it and I think it is fair to consider AI not so much a new block of interactive content but to see it as a new medium, a way to access information. It got jumpstarted by violating copyright on a massive scale (violating copyright law is what will get individuals into the poorhouse and will get companies very large valuations). But once that was done and the obvious content mountains exhauste the race was on to mine the remainder.

Content that was stored off-line or behind closed access (what are the chances that the likes of Google and Microsoft will be able to resist the lure of your mailbox and your file storage?) and content that was stored in inconvenient forms (books, for instance) all got digitized and gobbled up. The end of that is in sight as well.

But there is one source of fresh, and mostly un-tainted content that will remain: the direct interaction with the AI’s themselves. And this is where the future lies. I strongly believe that endless re-training AI on the conversations it has with actual humans is what will give the AI leaders their edge. This is content that only they have and that gives them a strong signal to help fix their confidence issues and the hallucination problem. Humans are correcting AI, effectively giving very high grade labeled training data every day. And lots of other useful information as well. By moving from a batch learning model with a release every so many months to a continuous learning model where your conversations are first stored as a shell around the original AI release and then later become incorporated into the inner core we will most likely end up improving the AIs we interact with the most.

It’s funny because we have to pay to access the AI which recycles a lot of stolen content and then we have to pay again to be allowed to teach it! Effectively the AI will become a crossbar between the brains of all of the people interacting with it and the closer that gets to real time the closer we will be to an actual hive mind. And deciding to ‘opt out’ will at that point probably make you effectively unemployable, you will be either augmented and expected to deliver or you will be on the dole.

The next decade will be interesting, and if there one thing I’m grateful for then it is that this wave hits me this late in my career, mostly because I wonder if there will still be room for dinosaurs like me, who refuse to have anything to do with AI in a professional setting. No matter how much Google wants to ram it down my throat.

AGI

Another often seen theme is that of AGI, the bigger brother of today’s systems. There is this hope amongst AI proponents that it will be the new frontier, super human intelligence. But I don’t think that that will matter as much. People are already using these systems as though they are AGI, and in that sense the narrow academic or techno definition is irrelevant. What matters is how people perceive it and when you look around you and see people talking to AIs about what they should cook for dinner and relationship advice then it is clear that even if AGI never actually happens it isn’t going to be a blocker to further adoption. We have to be careful that we’re not accidentally creating a new religion as well, the AI worshippers. I think I’ve already met some.

tldr: A small i386 32-bit protected mode multi-tasking operating system modeled after QNX

–

What is Unite?

Unite is an operating system in which everything is a process, including the things that you normally would expect to be part of the kernel. The hard disk driver is a user process, so is the file system running on top of it. The namespace manager is a user process. The whole thing (in theory, see below) supports network transparency from the ground up, you can use resources of other nodes in the network just as easily as you can use local resources, just prefix them with the node ID. In the late 80’s, early 90’s I had a lot of time on my hands. While living in the Netherlands I’d run into the QNX operating system that was sold locally through a distributor. The distributors brother had need of a 386 version of that OS but Quantum Software, the producers of QNX didn’t want to release a 386 version. So I decided to write my own. Luck had it that I ended up in Poland and that the lack of work there meant I could spend a lot of time on this (life was super cheap there back then compared to my savings). In the space of about a year I wrote a proof-of-concept kernel using DJGPP and a very kludgy development system that required me resetting the system after every failure. And there were lots of failures. Thousands of them, until, finally, one day I had a very rudimentary kernel and an ‘idle’ appliation running on top of it.

The next couple of years, until February 1994 (right after my first child was born and I decided I should probably get serious about earning money) I steadily improved on the original until it had a full complement of tools, was self hosting and came with a window manager and some graphics demos.

Why release this now?

When Linux was still very young, in 1992, there was a now infamous dispute between Andrew Tanenbaum, of MINIX fame and Linux Torvalds. The stakes were high: Tanenbaum thought Linus was wasting his time and talent building yet another macro kernel, he argued that micro kernels - even with a speed disadvantage - still have an edge in reliability and in speed of development. I agreed with Tanenbaum, but did not agree that MINIX was the way to go, I’d already seen QNX, and worked on my own variation on that theme and thought it was lightyears ahead of MINIX.

For the longest time this project sat on my hard drive in a state in which it could not run. The hardware that it had originally been developed on had died and I had never seen fit to spend more time on it to keep it bootable. Until the start of the fall of 2025, a good 30 years later, when I talked about it with my colleague I had not thought about putting any effort into a revival. In the meantime, a lot has happened. 32 bit systems are now already past, we’re on 64 bits now and 32 bit systems are usually doing only embedded stuff. We also have virtual machines, a luxury that my 30 year prior self would have happily traded a limb for, this makes development of such low level code a lot simpler than having to dive under the table every three minutes to hit the reset button. At some point this became such a routine that I hooked up a sustain pedal footswitch so I didn’t have to dive under the desk all the time.

So with modern tooling and the arena no longer vying for ‘the next best desktop solution’ I figured I should try to see if there is any life in it. Micro Kernels are fun to hack on: everything except for the core message passing task and scheduler is a user mode program and you can change it as easily as you can change any other program. That makes them unique and I think that this kind of hacking is now, especially with the more powerful embedded platforms within reach of just about anybody. The fact that it is a real time OS makes it much easier to work on hardware that requires guaranteed response time from the OS, there are 16 different priority levels and they offer very fine grained control over which tasks can be interrupted by which others.

Reviving an OS that hasn’t run in 30 years

This was a bit of a challenge. It took about two weeks, there were a couple of lucky breaks. The first was that I had not really done any cleaning of the repository where all this code was stored, so I had quite a few binaries that could run if I had a working OS. The second was that there was an image of a boot floppy. Between those two I managed to get the system booted under VirtualBox using a RAM disk. Some days later I had a working LBA capable hard drive, which I then populated with the contents of the ramdisk, and from there it was relatively quick work to recover the rest of the system.

The part that still eludes me is the bootfs.com program, this is the filesystem aware portion of the bootloader, that is loaded by the first stage on the boot floppy. There is some magic in there that for some reason does not play nice with the recovered version of TurboC, there is a chance I’m simply doing something wrong but I have not been able to successfully recompile that program and to get it to work so for now the system still boots from that original floppy image. But that’s fine, it’s not real hardware anyway and I’m sure if you really want to you can make it work again, all the pieces are there, I’ve added a VM disk image that should allow you to work on bootfs as well using freedos and some old binaries. Bootfs.com is a 16 bit real mode tiny memory model file that shuttles back-and-forth between real and protected mode to load all of the operating system components. It could boot from the hard drive as well (there is a ‘boot’ program to help you install the bootloader on an image) and like that you could do away with the floppy drive.

What does it look like?

Well, it looks like any old boring text mode operating system. There is the usual complement of utilities, a C compiler (and a C++ compiler), both are ancient but they work. There is the source tree in /usr/kernel.src and /usr/src as well as the usual unix like locations for binaries and such. There is a shell. There is some networking support but - again - I have not been able to resurrect this, and I can’t get the emulated COM ports to work either, which would have been nice as a way to move data in and out (for instance, using SLIP).

Quick start user guide

Install QEMU. Download the distro (see below). Unzip the distro file, and cd to unite/os, after that run ./unite.sh and you should see a virtual machine boot up in text mode. After it has booted and you’ve clicked in the VM windows to give it keyboard focus hit alt-f2 to move to another console (one where you can type commands) and type ‘cd 3:/’ to move to the hard drive. If you’re used to some variety of Unix you will feel mostly at home but note that ‘cli’ (command line interpreter) is not a regular shell. Interesting bits to look at are in 1:/boot/config.0 and of course the directory tree of drive 3:/ (the simulated hard disk). The file system is a very much stripped down version of the MINIX file system as it was at the time, it was meant to be replaced but I never got around to it. If you type ‘tsk’ you can see which tasks are currently running and ‘sac’ gives you a rudimentary system activity monitor. If you’ve used QNX at all in the past you will feel right at home.

Unite does not have the concept of a user, it is very much a single user OS but that single user can have multiple sessions. There is also a graphics mode, you can activate it by changing the ‘mode’ line in 1:/config/config.0 to ‘mode 16’ but I’d advise against that until you have the COM ports working and are able to type commands through there so you can see what you’re doing. If you are really adventurous: make a backup of the fda.img file; change that mode line and reboot. You’ll end up in a completely blind graphics mode. You should then type the following (all blind so you won’t know if you did it right or not until it works): ‘cd 3:/’ ; ‘cd bin/gp’ ; ‘ontty /dev/console.4 gp’ ; ‘ontty /dev/console.4 qwm’ ; ‘ontty /dev/console.4 qwmdesk’ (now the screen should change) ; ‘ontty /dev/console.4 qwmcalc’ (now you should see a small calculator, it would work if you had mouse/keyboard support but you don’t) ; ‘ontty /dev/console.4 qwmline’ (which should show a bouncing line demo).

Now you have a problem: if you reboot the system it will go right back to that graphics mode. So now you want to restore that fda.img backup you made above.

The Distribution

The operating system is distributed as a torrent:

magnet:?xt=urn:btih:bb01f5651944948b1b705ae4efa5ab5578bdee00&dn=unite.zip

The torrent contains a single zip file that you unpack, then you follow the instructions above.

Some notes:

• I have only tested this under Linux using VirtualBox and Qemu, I have not ran it under a Windows VM or on real hardware and you may run into issues if you do.

• there is no networking, this is probably the first thing that really needs doing

• the COM ports don’t work yet (this seems to be an emulation issue, I have yet to see a single byte in or out)

• the mouse driver (mp) does not work (it relies on a mouse plugged into a COM port, see above)

• the version of vi that is in /bin seems to add random cruft to files, I have not been able to figure out yet what the reason for that is.

• there is an alternative editor ‘e’ that does work, and that I wrote back in the stone age

• this is not/was not an effort at making something that is ‘secure’, if you plan on using this be very much aware of that and that you are most likely going to open yourself up to being compromised within seconds if you hook this up to the big bad internet. I was already very happy that I got as far as I did. But the Micro Kernel concept lends itself well to modification and you can start hacking on the kernel pretty much immediately, it is that simple.

• it is i386 protected mode only right now if you want another CPU or 64 bit then you have a lot of (fun!) work ahead of you.

• I have plenty to do and will not be available to manage this project or to even administer it, the source code is my full and only contribution, from here on in if you want to do something with it you are essentially on your own (or with your friends!), I have a lot of other stuff on my plate that takes precedence, so apologies for that but I still hope you find this useful.

License:

There are a lot of pieces that have been cobbled together from various other distributions, these were all made available at the time so in theory they should be fine to use but I have not verified whether any of those licenses have since been retracted or modified. Whichever party contributed the code still has the rights and this release is under the exact same license as under which the code was originally released in the early 90’s. The remaining code, the part of the code that I lay claim to (the core OS, the editor and the graphics library) I herewith place in the public domain, you can do with it whatever you want, as long as you credit the project and include a link to this page so others can start off from the same point. If you release an interesting derivative let me know and I’ll add a link to this page.

Credits:

• Henri Groenweg wrote the memory manager
• Marcel Wouters wrote the console and display driver
• DJGPP, by DJ DeLorie was the key to being able to start this at all
• QNX served as a very useful model and inspiration
• MINIX, provided in source code made it possible to have a file system; so thank you Andrew Tanenbaum, this made a massive difference

Screenshots

A short video showing the graphics mode