GPU SCAMMER Alert
A friend of mine, an immigrant to the Netherlands just got scammed. Someone offered a GPU for
sale and he committed to buying it. They agreed to meet and money changed hands, the guy
jumped on an e-bike and ran off. It turned out that the GPU and the VRAM had been removed
from the card making it worthless, leaving my friend with a 3000+ euro problem. He’s a student
in the AI/Robotics field so for him this is a very large amount of money to lose.
He went to the police to file a report but it turned out that all of the data the scammer
had provided was fake. My friend did have one ace up his sleeve, he had made a picture of
the person that was dropping off the card. So if you come across this character: Do not do
business with them. The platform this happened on was the Dutch website marktplaats, a local
marketplace that is owned by Ebay. If you have any data on this scammer I’d be more than happy
to forward it to my friend and to the police. The picture was taken near Zwolle train station.
Now, obviously this is very annoying for my friend, but the main reason I’m posting this is
that I want to warn anybody that is shopping around for GPUs right now:
The market is crazy. There are plenty of scams, and there are people unloading stolen
property. Treat any kind of deal with great care, don’t deal in cash, require to see
identification and only deal with people whose house address you’ve verified. It’s
a complete wild west and it will continue that way until prices normalize again, which
could be a while.
The Problem
There is a problem in the ‘infosphere’. In a nutshell: we now have high quality text that has low, zero or even negative value for the reader. Authorship is no longer something you can trust implicitly, even science is not immune to this and the problems seems to be set to become significantly larger in the near future. You could solve this by mandating authors label their work product but if there is money to be made by mis-labeling then you can expect people to do so.
Wikipedia has been fighting this trend for many years, they position themselves as a tertiary source: they will require references from secondary sources and those secondary sources should report on primary sources. This serves as an input filter of sorts because the assumption that secondary sources are going to do their homework was a good one. Until now.
Secondary sources are now incentivized to no longer do their homework. Just about every secondary source now uses automatic ways to generate content where it seems quantity, rather than quality has become a driving force. This is the equivalent of replacing quality home cooked food with fastfood: it is much easier to produce low quality food in quantity than it is to produce high quality food in quantity, if it can be done at all.
AI is a force multiplier here: it allows for the generation of immense quantities of very good looking text but with low quality. The information diet equivalent of fast food. And unlike fastfood, which usually is at least nourishing even if it isn’t the best of foods, the AI work product as a rule is poisonous: it contains ‘hallucinations’, which when allowed to take root can cause false information to spread widely. Now, it could be argued that fast food is when taken in quantity also poisonous so maybe the analogy is stronger than I intend for it to be. But the fact is that a single hamburger is not going to kill you. But a single falsehood just might, especially when the subjects are health, technology, engineering and so on.
Of course ‘unassisted humans’ also make mistakes. But we have ways to deal with that: reviews and editors. At the quantity of information produced that is manageable. But with the volume of automatically generated text exploding and absolutely dwarfing our combined work product it is a matter of time before the ‘slop’ will outnumber the combined human product by a considerable margin. Humans won’t be able to review or edit any of that simply because there wouldn’t be enough humans to do and besides the whole point seems to be to not employ all of these humans in the first place.
A possible solution
There are a couple of solutions but they are all going to rely to some degree on collaboration, ethics and honesty, both corporate and individually. If the advertising industry has taught us something then I guess that we do not have to hope for this to be successful. And bad actors will be able to put one over on good actors by gaining an unfair advantage. And since we only seem to want to punish people who gain an unfair advantage in life by clobbering old ladies over the head to make off with their handbags I have little hope that this will succeed. But I still think it is worth trying because the idea of giving up what we’ve achieved so far seems to me to be too high a price to pay.
AI - obviously - has its place. It can help us to have a personal teacher - who you can’t quite trust not to pull the wool over your eyes - with access to a vast amount of information. But as soon as you hit the ‘prompt / generate / cut / paste / profit’ cycle you are no longer operating from the same playbook as the rest of humanity: you are passing off the AIs work product as your own, whereas all you did was to create the prompt. And that’s fine, but then you should at least be transparent about it.
In order to help AI companies to create better products it would be good if we could create the digital equivalent of a sworn statement of authorship. I can see several ways of doing this and one in particular has my fancy because (1) it scales and (2) it would be very easy to implement.
Classification
Let’s start off by grading the different kinds of works
Original works. These are 100% human created, there is absolutely no question about the provenance of the text. We know who the author is, the author has made a clear statement of provenance and there is no reason to doubt that statement.
Assisted works. These are hybrids: the author has put considerable work into the text but has used AI to either stylistically improve the work or because some of the text has been AI generated. Ideally that would be a minority, as soon as you cross the 50% mark you will be in the next class.
AI generated, human augmented. These are hybrids too, but from the other side. Now the balance has shifted to majority AI input and minority human.
AI generated. 100% AI generated, no human input required.
One way to be able to distinguish what is human written and what is AI written would be by extending the Unicode character set with a complete duplicate and to legally mandate that all AI companies are only allowed to use this character set for their output. Transliteration to the human set would be strictly forbidden and should result in some civil penalty. Shipping an AI model that does not output in this transliterated set would result in a criminal penalty.
That’s harsh, yes. But I think that there is enough at stake here that this kind of measure is more than warranted. And AI companies have a huge incentive to play ball here: for one their execs would likely not want to be found engaging in criminal acts, for another they would be able to avoid that one thing that they’re all struggling with: to avoid eating their own (and each other’s) tail. Because AIs trained on slop will eventually degrade to the point where progress will stall or even reverse. Pre-AI text will have roughly the same value as low radiation steel (https://en.wikipedia.org/wiki/Low-background_steel). By labeling their output in this way AI companies will be able to sift out their own and other AI companies’ output.
Finally, a human run attestation service could be run: a sufficiently secure cryptographic hash run on a piece of input text that measures the ratio of AI vs human generated text in the input could give a meta label that can easily be affixed to the content. Search engines and AI ingestion could then use these marks to differentiate between various sources and hopefully use this as a degree of measuring quality and giving higher quality results priority. So you don’t accidentally end up biting into fast food when you were expecting something much better.
Problems with the above
Of course, it is easy to dismiss this all out of hand because it isn’t - and won’t be - perfect. I can already see many sub-problems that will need to be resolve and the knee-jerk reaction will - of course - be that the race is already run and it is much too late to do something about it now so we may as well lie down and be run over by the bulldozer.
But I’m an optimist by nature: I think we can recognize that even if this plan is flawed the spirit of it has merit, fix the flaws and then implement them to the best of our ability. It’s the spirit that counts here, not the letter, so let your imagination loose on taking the idea and improving on it rather than to bash it with the first middlebrow dismissal that comes to mind.
–
This text was written by hand, using a brain and a computer in the most passive way possible, to record keystrokes. No automatic processes contributed to the content. AI percentage: 0.
I’m impressed
For the last couple of years, my colleagues and myself have been playing around with AI. I can’t speak for them but from the perspective of a casual user I am very much impressed by the capabilities on display. If you dedicate some time to it you can get a lot of mileage out of AI, it can help you in the way a highly interactive book would help, with vast amounts of knowledge at the tips of your fingers. As long as you are wary of the inevitable hallucinations and the AI’s tendency to get stuck in loops. Because the one thing that we have not yet managed to program in there is the ability to be humble: to realize that no matter how much knowledge you’ve got you don’t know everything. So “I don’t know” is not on the menu. It is not the kind of thing I have seen as output from the various AI offerings.
If you had told me ten years ago that we would be having conversations with a machine at this level today I would have thought it to be impossible, and yet, here we are. The ‘Attention is all you need paper’ combined with Hinton’s persistence paid off immensely, probably more than even the people behind that effort had expected. Current generative AI is able to converse on an immense variety of topics at a reasonably high level.
Professional use
Even so, I would never use its output in a professional setting. When we are hired to do work we are hired to do work, not to come up with prompts and then to let the AI do the work. Anybody can do that, it would feel like fraud to use AI as some kind of shortcut, then we may as well give our customers the prompts.
As a search engine and uniform interface to documentation though, it is unparalleled, as long as you check the output for factual correctness.
Commercializing AI
Because we like playing around with new tech we put together a monster computer with a lot of GPUs and a very large amount of memory. This gives us the capability to run these models in-house and to evaluate their abilities without accidentally leaking our data to the AI providers. What struck me is that in any conversation longer than a couple of lines you end up with one or more times where you have to correct the AI on some point, after which the conversation may stay on the rails a bit longer. But there is always a point where the growing context becomes problematic and of course when you start a new session all of those corrections are forgotten. That plus any kind of formatting instructions regarding the responses have to be repeated over and over again at the beginning of every new session, a kind of ‘system prompt’ at the user level. Running a system like this commercially requires a lot of power, memory, computation and storage for every a single user. I’m fortunate in that I have a lot of solar panels but if not for that my power bill would be astronomical.
And that’s where the data centers and the price of RAM and GPUs comes in. The price of a stick of DDR5 memory must have the smell of onions baked in because when I look I start crying. This will force the AI companies to commercialize it but their avenues to do so are limited, and I’m not sure how they think they are going to survive without doing some kind of rug-pull.
Confidence
What bugs me about the interaction with the various AI models is that they are so incredibly over confident. They are jump-to-conclusion machines that will answer a one line prompt with six pages of plausible nonsense as often as they get it right. The signal-to-noise ratio can be very low and if you’re not careful you will end up reading a lot of self congratulationary bullshit that does not move the needle at all in terms of your understanding or insight. The number of times I’ve seen the word ‘working’ for something that could not work and ‘no handwaving’ followed by a lot of handwaving is pretty high, at least as high as the number of times that I came away with the idea that I had genuinely learned something. It would be great if the current crop of AI used font size or levels of gray to denote the things they are actually confident about and the things that are speculative. This alone would significantly increase its value to me.
When it does not work
The most use I seem to get out of it is when I’m studying some subject that I already know the basics of to increase that knowledge and give hints on fruitful avenues of reading or research. It rarely happens that the interaction with the AI itself gives direct insight on things that are non-obvious. Over the course of the year that has happened probably a handful of times. The more clueless I am about a subject the less use I get out of AI. Conversely, the more I know about something the more I’m irritated at the basic stuff that it - repeatedly - gets wrong and you end up with endless corrections. It is able to pull a massive amount of information up at a moments notice but it will happily forget something that came up only a few minutes ago. This can be quite annoying, but then, whenever I’m annoyed I am also impressed: I am annoyed at a machine for not being able to so something as if it were a person. And it clearly is not, despite all of the language hints that it is.
The Sleaze
Some of the models that are in widespread use are extremely slick at giving the user the impression that they are smart. They will endlessly congratulate and reinforce that the user is a smart person even when they’re clearly not. The paternalizing attitude is so ingrained that even after multiple reminders it will often just find new ways to effectively do the same thing.
The Hive Mind
After playing with something for a while I always wonder: What’s next? Where are we headed? I think the current crop of AI products is useful, but the push towards commercialization will most likely destroy a lot of that utility. The same has happened with all other media that preceded it and I think it is fair to consider AI not so much a new block of interactive content but to see it as a new medium, a way to access information. It got jumpstarted by violating copyright on a massive scale (violating copyright law is what will get individuals into the poorhouse and will get companies very large valuations). But once that was done and the obvious content mountains exhauste the race was on to mine the remainder.
Content that was stored off-line or behind closed access (what are the chances that the likes of Google and Microsoft will be able to resist the lure of your mailbox and your file storage?) and content that was stored in inconvenient forms (books, for instance) all got digitized and gobbled up. The end of that is in sight as well.
But there is one source of fresh, and mostly un-tainted content that will remain: the direct interaction with the AI’s themselves. And this is where the future lies. I strongly believe that endless re-training AI on the conversations it has with actual humans is what will give the AI leaders their edge. This is content that only they have and that gives them a strong signal to help fix their confidence issues and the hallucination problem. Humans are correcting AI, effectively giving very high grade labeled training data every day. And lots of other useful information as well. By moving from a batch learning model with a release every so many months to a continuous learning model where your conversations are first stored as a shell around the original AI release and then later become incorporated into the inner core we will most likely end up improving the AIs we interact with the most.
It’s funny because we have to pay to access the AI which recycles a lot of stolen content and then we have to pay again to be allowed to teach it! Effectively the AI will become a crossbar between the brains of all of the people interacting with it and the closer that gets to real time the closer we will be to an actual hive mind. And deciding to ‘opt out’ will at that point probably make you effectively unemployable, you will be either augmented and expected to deliver or you will be on the dole.
The next decade will be interesting, and if there one thing I’m grateful for then it is that this wave hits me this late in my career, mostly because I wonder if there will still be room for dinosaurs like me, who refuse to have anything to do with AI in a professional setting. No matter how much Google wants to ram it down my throat.
AGI
Another often seen theme is that of AGI, the bigger brother of today’s systems. There is this hope amongst AI proponents that it will be the new frontier, super human intelligence. But I don’t think that that will matter as much. People are already using these systems as though they are AGI, and in that sense the narrow academic or techno definition is irrelevant. What matters is how people perceive it and when you look around you and see people talking to AIs about what they should cook for dinner and relationship advice then it is clear that even if AGI never actually happens it isn’t going to be a blocker to further adoption. We have to be careful that we’re not accidentally creating a new religion as well, the AI worshippers. I think I’ve already met some.
tldr: A small i386 32-bit protected mode multi-tasking operating system modeled after QNX
–
What is Unite?
Unite is an operating system in which everything is a process, including the things that you normally would expect to be part of the kernel. The hard disk driver is a user process, so is the file system running on top of it. The namespace manager is a user process. The whole thing (in theory, see below) supports network transparency from the ground up, you can use resources of other nodes in the network just as easily as you can use local resources, just prefix them with the node ID. In the late 80’s, early 90’s I had a lot of time on my hands. While living in the Netherlands I’d run into the QNX operating system that was sold locally through a distributor. The distributors brother had need of a 386 version of that OS but Quantum Software, the producers of QNX didn’t want to release a 386 version. So I decided to write my own. Luck had it that I ended up in Poland and that the lack of work there meant I could spend a lot of time on this (life was super cheap there back then compared to my savings). In the space of about a year I wrote a proof-of-concept kernel using DJGPP and a very kludgy development system that required me resetting the system after every failure. And there were lots of failures. Thousands of them, until, finally, one day I had a very rudimentary kernel and an ‘idle’ appliation running on top of it.
The next couple of years, until February 1994 (right after my first child was born and I decided I should probably get serious about earning money) I steadily improved on the original until it had a full complement of tools, was self hosting and came with a window manager and some graphics demos.
Why release this now?
When Linux was still very young, in 1992, there was a now infamous dispute between Andrew Tanenbaum, of MINIX fame and Linux Torvalds. The stakes were high: Tanenbaum thought Linus was wasting his time and talent building yet another macro kernel, he argued that micro kernels - even with a speed disadvantage - still have an edge in reliability and in speed of development. I agreed with Tanenbaum, but did not agree that MINIX was the way to go, I’d already seen QNX, and worked on my own variation on that theme and thought it was lightyears ahead of MINIX.
For the longest time this project sat on my hard drive in a state in which it could not run. The hardware that it had originally been developed on had died and I had never seen fit to spend more time on it to keep it bootable. Until the start of the fall of 2025, a good 30 years later, when I talked about it with my colleague I had not thought about putting any effort into a revival. In the meantime, a lot has happened. 32 bit systems are now already past, we’re on 64 bits now and 32 bit systems are usually doing only embedded stuff. We also have virtual machines, a luxury that my 30 year prior self would have happily traded a limb for, this makes development of such low level code a lot simpler than having to dive under the table every three minutes to hit the reset button. At some point this became such a routine that I hooked up a sustain pedal footswitch so I didn’t have to dive under the desk all the time.
So with modern tooling and the arena no longer vying for ‘the next best desktop solution’ I figured I should try to see if there is any life in it. Micro Kernels are fun to hack on: everything except for the core message passing task and scheduler is a user mode program and you can change it as easily as you can change any other program. That makes them unique and I think that this kind of hacking is now, especially with the more powerful embedded platforms within reach of just about anybody. The fact that it is a real time OS makes it much easier to work on hardware that requires guaranteed response time from the OS, there are 16 different priority levels and they offer very fine grained control over which tasks can be interrupted by which others.
Reviving an OS that hasn’t run in 30 years
This was a bit of a challenge. It took about two weeks, there were a couple of lucky breaks. The first was that I had not really done any cleaning of the repository where all this code was stored, so I had quite a few binaries that could run if I had a working OS. The second was that there was an image of a boot floppy. Between those two I managed to get the system booted under VirtualBox using a RAM disk. Some days later I had a working LBA capable hard drive, which I then populated with the contents of the ramdisk, and from there it was relatively quick work to recover the rest of the system.
The part that still eludes me is the bootfs.com program, this is the filesystem aware portion of the bootloader, that is loaded by the first stage on the boot floppy. There is some magic in there that for some reason does not play nice with the recovered version of TurboC, there is a chance I’m simply doing something wrong but I have not been able to successfully recompile that program and to get it to work so for now the system still boots from that original floppy image. But that’s fine, it’s not real hardware anyway and I’m sure if you really want to you can make it work again, all the pieces are there, I’ve added a VM disk image that should allow you to work on bootfs as well using freedos and some old binaries. Bootfs.com is a 16 bit real mode tiny memory model file that shuttles back-and-forth between real and protected mode to load all of the operating system components. It could boot from the hard drive as well (there is a ‘boot’ program to help you install the bootloader on an image) and like that you could do away with the floppy drive.
What does it look like?
Well, it looks like any old boring text mode operating system. There is the usual complement of utilities, a C compiler (and a C++ compiler), both are ancient but they work. There is the source tree in /usr/kernel.src and /usr/src as well as the usual unix like locations for binaries and such. There is a shell. There is some networking support but - again - I have not been able to resurrect this, and I can’t get the emulated COM ports to work either, which would have been nice as a way to move data in and out (for instance, using SLIP).
Quick start user guide
Install QEMU. Download the distro (see below). Unzip the distro file, and cd to unite/os, after that run ./unite.sh and you should see a virtual machine boot up in text mode. After it has booted and you’ve clicked in the VM windows to give it keyboard focus hit alt-f2 to move to another console (one where you can type commands) and type ‘cd 3:/’ to move to the hard drive. If you’re used to some variety of Unix you will feel mostly at home but note that ‘cli’ (command line interpreter) is not a regular shell. Interesting bits to look at are in 1:/boot/config.0 and of course the directory tree of drive 3:/ (the simulated hard disk). The file system is a very much stripped down version of the MINIX file system as it was at the time, it was meant to be replaced but I never got around to it. If you type ‘tsk’ you can see which tasks are currently running and ‘sac’ gives you a rudimentary system activity monitor. If you’ve used QNX at all in the past you will feel right at home.
Unite does not have the concept of a user, it is very much a single user OS but that single user can have multiple sessions. There is also a graphics mode, you can activate it by changing the ‘mode’ line in 1:/config/config.0 to ‘mode 16’ but I’d advise against that until you have the COM ports working and are able to type commands through there so you can see what you’re doing. If you are really adventurous: make a backup of the fda.img file; change that mode line and reboot. You’ll end up in a completely blind graphics mode. You should then type the following (all blind so you won’t know if you did it right or not until it works): ‘cd 3:/’ ; ‘cd bin/gp’ ; ‘ontty /dev/console.4 gp’ ; ‘ontty /dev/console.4 qwm’ ; ‘ontty /dev/console.4 qwmdesk’ (now the screen should change) ; ‘ontty /dev/console.4 qwmcalc’ (now you should see a small calculator, it would work if you had mouse/keyboard support but you don’t) ; ‘ontty /dev/console.4 qwmline’ (which should show a bouncing line demo).
Now you have a problem: if you reboot the system it will go right back to that graphics mode. So now you want to restore that fda.img backup you made above.
The Distribution
The operating system is distributed as a torrent:
magnet:?xt=urn:btih:bb01f5651944948b1b705ae4efa5ab5578bdee00&dn=unite.zip
The torrent contains a single zip file that you unpack, then you follow the instructions above.
Some notes:
I have only tested this under Linux using VirtualBox and Qemu, I have not ran it under a Windows VM or on real hardware and you may run into issues if you do.
there is no networking, this is probably the first thing that really needs doing
the COM ports don’t work yet (this seems to be an emulation issue, I have yet to see a single byte in or out)
the mouse driver (mp) does not work (it relies on a mouse plugged into a COM port, see above)
the version of vi that is in /bin seems to add random cruft to files, I have not been able to figure out yet what the reason for that is.
there is an alternative editor ‘e’ that does work, and that I wrote back in the stone age
this is not/was not an effort at making something that is ‘secure’, if you plan on using this be very much aware of that and that you are most likely going to open yourself up to being compromised within seconds if you hook this up to the big bad internet. I was already very happy that I got as far as I did. But the Micro Kernel concept lends itself well to modification and you can start hacking on the kernel pretty much immediately, it is that simple.
it is i386 protected mode only right now if you want another CPU or 64 bit then you have a lot of (fun!) work ahead of you.
I have plenty to do and will not be available to manage this project or to even administer it, the source code is my full and only contribution, from here on in if you want to do something with it you are essentially on your own (or with your friends!), I have a lot of other stuff on my plate that takes precedence, so apologies for that but I still hope you find this useful.
License:
There are a lot of pieces that have been cobbled together from various other distributions, these were all made available at the time so in theory they should be fine to use but I have not verified whether any of those licenses have since been retracted or modified. Whichever party contributed the code still has the rights and this release is under the exact same license as under which the code was originally released in the early 90’s. The remaining code, the part of the code that I lay claim to (the core OS, the editor and the graphics library) I herewith place in the public domain, you can do with it whatever you want, as long as you credit the project and include a link to this page so others can start off from the same point. If you release an interesting derivative let me know and I’ll add a link to this page.
Credits:
- Henri Groenweg wrote the memory manager
- Marcel Wouters wrote the console and display driver
- DJGPP, by DJ DeLorie was the key to being able to start this at all
- QNX served as a very useful model and inspiration
- MINIX, provided in source code made it possible to have a file system; so thank you Andrew Tanenbaum, this made a massive difference
Screenshots
A short video showing the graphics mode
tldr: limit your GPUs to about 2/3rd of maximum power draw for the least Joules consumed per token generated without speed penalty.
–
Why run an LLM yourself in the first place?
The llama.cpp software suite is a very impressive piece of work. It is a key element in some of the stuff that I’m playing around with on my home systems, which for good reasons I will never be able to use in conjunction with a paid offering from an online provider. It’s interesting how the typical use-case for LLMs is best served by the larger providers and the smaller models are just useful to gain an understanding and to whet your appetite. But for some applications the legalities preclude any such sharing and then you’re up the creek without a paddle. The question then is: is ‘some AI’ better than no AI at all and if it is what does it cost?
What to optimize for?
And this is where things get complicated very quickly. There are so many parameters that you could use to measure your systems’ performance that it can be quite difficult to determine if a change you made was an improvement or rather the opposite. After playing around with this stuff a colleague suggested I look into the power consumption per token generated.
And that turned out - all other things being equal - to be an excellent measure to optimize for. It has a number of very useful properties: it’s a scalar, so it is easy to see whether you’re improving or not, it’s very well defined: Joules / token generated is all there is to it and given a uniform way of measuring the property of a system you can compare it to other systems even if they’re not in the same locality. You can average it over a longer period of time to get you reasonably accurate readings even if in the short term there is some fluctuation.
You could optimize for speed instead but for very long running jobs (mine tend to run for weeks or months) speed is overrated as a metric, but for interactive work it is clearly the best thing to optimize for as long as the hardware stays healthy. And ultimately Joules burned per token is an excellent proxy for cost (assuming you pay for your energy).
Creating the samples
So much for the theory. In practice though, measuring this number turned out to be more of a hassle than I thought it would be. There are a couple of things that complicate this: PCs do not normally come with easy ways to measure how much power they consume and LLama doesn’t have a way to track GPU power consumption during its computations. But there are some ways around this.
I started out by parsing the last 10 lines of the main.log file for every run of llama.cpp. That works as long as you only have one GPU and run only one instance. But if you have the VRAM and possibly multiple GPUs this will rapidly become a hot mess. The solution is to ensure that every run pipes its output to a unique log file, parse the last couple of lines to fish out the relevant numbers and then to wipe the log file.
Sampling the power consumption was done using a ‘Shelly’ IP enabled socket, it contains a small webserver that you can query to see how much power is drawn though the socket. It’s not super precise, secure or accurate but more than good enough for this purpose. Simply hit
http://$LANIPADDRESS/meter/0
and it will report the status of the device as a json object, which is trivial to parse. Nvidia’s nvidia-smi program will show you the status of the GPUs and report their current power draw. Subtract that out and you’re left with just the CPU, RAM, MB and the drives.
Experiment
You then let this run for a while to give you enough samples to draw conclusions from. In order to figure out what the sweet spot of the setup I have here is (right now 2x 3090, i7) I stepped through the power consumption of the GPUs in small increments. 25 Watts per step, with a multi-hour run to get some precision in the measurements. Nvidia cards have an internal power limiter that allows you to set an upper limit to their consumption in Watts for a dual GPU setup the command line is:
nvidia-smi –power-limit=$Wattage –id=0,1
Just replace $Wattage with whatever limit you desire, but keep it above 100 and below the maximum power draw of the GPU for meaningful results.
Doing this for a series of steps using the Llama 13B 5 bit sampled model (which uses about 10G of VRAM), running four instances of llama.cpp concurrently (so two on each card) for an hour per step to reduce sampling inaccuracy gives the following graph of power consumption limits of the GPUs (horizontal) vs Joules consumed per token generated (vertical):
In tabular form:
GPUs (W) tps J/token GPU Temp (W)
300 27 19.2 43
350 35 15.1 48
400 48 12.3 50
450 52 12.1 53
500 51 12.5 53
550 49 13.8 55
600 50 14.7 57
650 50 15.8 60
700 50 16.2 63
Conclusion
The ‘sweet spot’ is pretty clear, the graph starts off on the left at about 19 Joules per token because the simple fact that the machine is on dominates the calculation. Not unlike a car that is idling or going very slow: you make no progress but you consume quite a bit of power. Then, as you ramp up the power consumption resulting in more processing efficiency increases fairly rapidly, up to a point. And after that point efficiency drops again, this is because now thermal effects are dominating and more input power does not result in that much more computation. If you live in a cold climate you may think of this as co-generation ;) In an extreme case you might run into thermal throttling, which is because of the way llama.cpp handles the interaction between the CPU and GPU just going to lead to a lot of wasted power. So there is a clearly defined optimum, for this dual 3090 + i7 CPU that seems to lie somewhere around 225W per GPU.
The efficiency is still quite bad (big operators report 4-5 Joules / token on far larger models than these), but it is much better than what you’d get out of a naively configured system, resulting in more results for a given expense in power (and where I live that translates into real money). If you have other dominant factors (such as time) or if power is very cheap where you live then your ‘sweet spot’ may well be somewhere else. But for me this method helped run some pretty big jobs at an affordable budget, and it just so happens that the optimum efficiency also coincides with very close to peak performance in terms of tokens generated per unit time.
Caveats
Note that these findings are probably very much dependent on:
- workload
- model used
- the current version of llama.cpp (which is improving all the time) and the particular settings that I compiled it with (make LLAMA_FAST=1 LLAMA_CUBLAS=1 NVCCFLAGS=“–forward-unknown-to-host-compiler -arch=sm_86” CUDA_DOCKER_ARCH=“sm_86”)
- Hardware
- GPU (Asus ‘Turbo’ NV 3090)
- CPU (i7-10700K stepping 5)
- avoid thermal throttling (which would upset the benchmark)
- CPU and case fans set to maximum (noisy!)
- GPU fans set to 80%
- ambient temperature (approximately 24 degrees in the room where these tests were done)
- Quite possibly other factors that I haven’t thought of.
Please take these caveats seriously and run your own tests, and in general don’t trust fresh benchmarks of anything that hasn’t been reproduced by others (Including this one!) or that you yourself haven’t reproduced yet. If you have interesting results that contradict or confirm these then of course I’d love to hear about it via jacques@modularcompany.com, and if the information warrants it I’ll be more than happy to amend this article.