Hello COMET
This article is a work in progress.
It's been about a month since I started testing this tiny Autonomys Farmer capable of 16TB, and I'm honestly impressed.
I wanted to write an article on what the COMET is and why I have a whole section dedicated to them. In short, the COMET consists of a Custom 3D Printed Case + CM3588 (8G/64G) + NAS Kit (4x M.2 NVME, 2.5G NIC) + 60W PSU + Debian OS. The CM3588 gets some noteriety as it was featured by Linus Tech Tips in their Paying for Cloud Storage is Stupid video. Of course it can be used for a NAS, but my use case is focused on Crypto. Specifically, I plan to use 8 of these to farm on the Autonomys Network.
Testing on Autonomys
Regardless of what you think about Crypto, jamming 4x NVME into such a small form factor is pretty darn cool. Maxing it out with 4x NVME results each NVME running @ PCIe 3.0 x1. This is not a significant amount of bandwidth. However, Autonomys is random read IOPS heavy, not bandwidth heavy. In my testing with 4x 4TB NVMEs fully plotted, I did not have a single missed reward in my week long testing. This can vary based on a lot of factors, but I think it's safe to say PCIe 3.0 x1 will not be an issue at 16TB.
The 2.5G ethernet port allows for very quick plotting over your network, which is perfect for a dedicated Farmer like the COMET. I was not able to saturate the connection with 2x RTX 4090. However, it did appear that the cards were not being fully utilized. For most people, who will be plotting via a single GPU, this would be more than sufficient.
Thermals
With so much packed into such a tight space, thermals were a bit of a worry. The board comes without any active cooling (however the COMET does have a 5v 50mm dual ball bearing fan). The NVME slots are also VERY tight, so most NVME heatsinks do not fit. I was able to use the ones that are held on by rubberbands. While not pretty, they did over some improvements in use with the COMET as the vents of the case direct air over the NVMEs and CPU.
I recorded some data for both the CPU and the NVMEs as those were the biggest worry. The fan and heatsinks add quite a bit of effort as the 50mm fan requires soldering on the right connector and drilling out the holes for the screws I wanted to use, so I really wanted to see if all of that was worth it. It also adds about $15 in material costs per COMET.
In the below charts, this was done with only the Farmer running, and plotting at about 1 sector per minute. At the time of the tests I was having issues with GPU plotting which is why it was a bit slow. Each test ran for one hour.
No Active Cooling, no NVME Heatsinks
Active Cooling, NVME Heatsinks
No Active Cooling, NVME Heatsinks
Active Cooling, NVME Heatsinks
To really stress things I turned on CPU plotting for the COMET.
No Active Cooling, no NVME Heatsinks
Active Cooling, NVME Heatsinks
No Active Cooling, NVME Heatsinks
Active Cooling, NVME Heatsinks
Resource Usage
The next concern I had was if the CM3588 was going to have sufficient power. I really wanted to be able to run a fully encapsulated Autonomys Node + Farmer with a 8x COMET cluster, even if one of those was a dedicated Node. To do this I fully plotted two COMETs and created a few different setups. For all these tests I had the maximum cooling (Heatsinks + 50mm Fan)
Farmer Only
For the first test I am using COMET-Alpha as a dedicated Farmer. This is the least amount of resource usage.
Node Only
Next, I wanted to see what sort of stress running a Node on the COMET would provide. In order to get the full picture, I started with a fresh node, storing the Node Data on a USB mounted SSD.
Farmer + Plotter
Moving back to farming, I wanted to see how it would handle running both a Farmer and a Plotter. I never intended to perform initial plotting with these, so the NVMEs are fully plotted. My goal here is to see if these can handle replotting or if I will need to always have a dedicated GPU plotter hooked up to them.
Full Cluster
If I really want to have these self-plot, at least one of them will require a full Cluster which includes a Controller, Cache, and NATs. In this configuration, the COMET is once again fully plotted and will only be handling replots. The Cache is stored on a USB connected SSD and set to 110GiB which should be enough to hold a full cache.
Farming Cluster + Node
Finally, I brought in a second COMET to see how a fully self-reliant COMET cluster would perform. In order to spread the load as efficiently as possible I have the following configuration:
COMET Alpha - Node + Cluster Farmer COMET Bravo - Cluster Farmer, Cluster Cache, Cluster Controller, Cluster Plotter, NATs
My thought with this is that the Node is extremely resource intensive, so I will put as much of the Cluster components as I can on the other COMET to spread the load.