Systems Manual

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
=====Limulus Compute Blades=====

====Introduction====

The Limulus 2.0 Micro-ATX (μATX) blade represents a new generation of local, low-cost, and high-performance computing (e.g. for an office, lab, classroom, factory, a.k.a. edge computing). The modular design allows virtually any standard μATX motherboard to be combined into a dense high-performance computing resource using Limulus desk-side and rack-mount enclosures. The blade design provides an economic and flexible platform on which to build high performance solutions.  

{{ :wiki:figure-1-blade-above.png?400 |}}

**Figure One:** //The Limulus 2.0 Micro-ATX blade//

====History====
High-performance Limulus computing systems are designed to operate in "No Data Center Needed" environments. In addition, Limulus systems provide a cost-effective balance between commodity hardware pricing and high performance.  For this reason, systems are built with as many high-quality market-priced commodity components as possible. 

In order to condense the space needed for packing hardware into a commercial computing case, custom brackets and fittings were designed and manufactured using sheet metal. To keep the amount of expensive custom components to a minimum, motherboards were stacked and combined into a single unit (a Limulus Unit). While this approach was successful, it did not allow individual access to each motherboard. 

The advent of 3-D printing changed the whole approach of Limulus "No Data Center Needed" computing. Instead of purchasing minimum quantities of custom sheet metal parts, all the physical components can now be printed on-demand and altered without any economic penalty (i.e. sheet metal components purchased in quantity that no longer fit are wasted).

In addition to lowering cost and providing more flexibility, 3-D printing provided a means to manufacture complex components that would have been price-prohibitive using traditional "large quantity" manufacturing methods.  The Limulus 2.0 μATX blade system is the first of many new capabilities offered by 3-D printing.  

{{ :wiki:figure-2-motherboard-tray-combined.png?400 |}}

**Figure Two:** //3-D printed design for μATX motherboard blades. Separate front panel and IO plate allow for customization for different motherboards//

====Capabilities and Costs====
The following are some of the important capabilities of the Limulus 2.0 μATX blade system.

  * Support for all standard μATX commodity motherboards
  * Single socket supporting up to 80-watt (W) Thermal Power Design (TDP) processors 
  * Up to 128 Gigabytes (GB) of DDR4 memory with Error Correction Code (ECC) option
  * One or two onboard M.2 NVMe drives (depends on motherboard)
  * Two front-facing SATA ports (as eSATA connector)
  * Front-facing PCIe slot (used for 10 GbE option) 
  * All motherboard cabling on front (Video/USB/Network)
  * 12 Volt Hot Swap Pluggable Power Connector
  * Installable in Limulus desk-side cases or rack-mount systems.


Many capabilities depend on the motherboard model. For example, current offerings include AMD Ryzen processors (3.6 GHz) with eight physical cores, up to 64 GB of ECC memory, NVMe drive, and 10 GbE networking.  Depending on market pricing, the approximate starting cost of a Limulus 2.0 μATX blade is less than $750.00 (as of Q2-2020). This cost compares favorably with traditional data center equipment. 

{{ :wiki:figure-3-blade-front-annotated.png?400 |}}
**Figure Three:** //μATX blade front panel//

====Design Details====
All Limulus systems are based on a balanced design that incorporates low-cost commodity hardware and high performance.  Consult the [[Functional Diagram|Functional Diagram]] for design details. The design rationale for the various blade components is as follows:

===Edge Computing===
Since their inception, Limulus systems were designed to operate in a local or "non-datacenter" environment. As mentioned, this can be an office, lab, classroom, or factory. In general, this type of computing is now categorized as "Edge Computing," where processing occurs at the edge of the data center or cloud. The Limulus μATX blade is designed with the flexibility needed for this environment. Virtually all aspects of the computing node (processor, cores, memory, network, and power) can be configured to match Edge computing needs. 

===μATX Motherboards===
The use of μATX represents a good compromise between size and motherboard capabilities. There are other, smaller motherboard form factors (e.g. mini IDX) that require less space; however, these often do not provide as much flexibility in terms of processor choice, processor cooling options, supported chip-sets, amounts of memory, and PCIe slots. Larger, full ATX motherboards primarily have more PCIe slots, but almost all the features found on these larger motherboards can be found on an appropriate μATX system. In addition, since the blades rely on motherboard "bus benders" to turn one PCIe socket by 90 degrees, additional PCIe ports available on full ATX systems would not be usable due to space considerations.

===Use Low Core Count – High Clock Processors===
The trend in processor design is to increase the number of cores on the processors' die. While this may be advantageous for some workloads, the increased core count requires lower clock speeds to fit processors into an air-cooled TDP envelope (TDP air cooling gets difficult above 275 W).  For instance, a current-generation 32-core AMD EPIC has a base clock speed of 2.5 GHz and a TDP of 180 W.  

Limulus μATX blades take a different approach. Using lower core count processors with higher clock speeds (in a clustered configuration) provides a less costly and higher performance solution. As mentioned above, the current generation of AMD Ryzen processors offers eight physical cores running at a base speed of 3.6 GHz with a total TDP of 65 W. Lowering the core count allows the clock speed to increase by 44% over the slower EPIC processor. (Note: Base clock speed is used for comparisons because full or nearly full utilization of the blade will prohibit turbo or boost clock increases.) 

===Use Low Power Processors ===
Another advantage of low core count high clock processors is the lower TDP. For example, both Intel and AMD manufacture processors that fit within a 65-80 W TDP. Lower TDP means less heat and more relaxed cooling requirements (slower/quieter fans). In addition, less heat generation does not require data center cooling (e.g. hot aisle/cold aisle) and allows Limulus μATX blades to be used almost anywhere. 

In a 4-blade cluster environment, the cumulative TDP can reach 260 W; however, this heat is distributed, rather than emanating from a single location (i.e. using air cooling is more efficient to move four smaller amounts of heat vs. a large centrally-located heat source). 

===CPU Only===
The size and compactness of the Limulus μATX blade excludes the use of GPU acceleration. In addition, the thermal constraints described above do not allow for large 200-300 W GPUs to be placed on the blades.  The power requirements and heat generated by using multiple GPUs can limit where systems can be used.  

===On-Board Video===
Many of the applicable Intel desktop and server processors have onboard video. This video can be taken directly from the blade motherboard.  Under normal operation, this video feed is not used and amounts to a text terminal connection to the blade that can be accomplished via a network connection. 

On systems with AMD Ryzen processors, there is no onboard video option even though there may be video connectors on the motherboard. Under normal operation, this is not an issue because the blade can often boot the Linux operating system without a video card. As mentioned above, the video output only provides a text terminal connection to the blade. This connection can be conveniently made over the internal network. (Note: Booting Linux without a video card may be motherboard-dependent. This may eliminate some motherboards from operating in the blades.)

=== Memory and ECC===
Depending on the motherboard and processors, the maximum amount of memory may range between 64 GB and 128 GB (or higher) per blade. While this is much less than the traditional data center server, it does provide enough memory to run multiple and/or large distributed applications. In particular, if the blades are used in an HPC cluster running distributed Message Passing Interface (MPI) applications, the total available memory across four nodes is 512 GB and can reach 1 TB using eight nodes in the large Limulus case.

In terms of ECC memory, support by motherboard and processor varies. In general, Intel i5 and i7 processors do not provide this capability, while AMD does (although AMD has stated they have not qualified ECC memory on Ryzen processors). There is also some discussion about the practical importance of ECC memory (See: https://www.clustermonkey.net/EDGE-Computing-Series/if-i-don-t-use-ecc-have-i-made-an-error.html).

All Limulus μATX that use AMD processors are equipped with fully-qualified ECC memory (i.e. Internal tests were performed to confirm memory errors were corrected on these platforms).

Intel does offer Xeon-based processors that can operate on Limulus μATX blades and provide ECC memory. The cost of these blades is somewhat higher and the TDP ratings are around 80 W.

===Networking and PCIe Slots===
Blades offer several networking options. First, virtually all μATX motherboards provide at least one 1 GbE port. There are other motherboards that provide two 1 GbE ports or one 10 GbE port. These ports are almost always "Base-T" connectors (plastic "click in" type) using CAT 6 or 6A cabling. In addition to the motherboard ports, Limulus μATX blades can bring one of the PCIe slots out to the front of the blade. This slot is normally used for a 10 GbE adaptor (either SFP+ or Base-T), although many types of smaller single slot PCIe cards can be used in this position. The 3D-printed front of the blade can be modified to accept different PCIe cards with different physical port locations and types.

====Physical Configuration/RAS ====
Blades are easily inserted (or removed) from the standard Limulus case. Power is applied through a blind 12V power connector in the rear of the blade. All other connections are brought to the front of the blade (see Figure Three). These connections include all motherboard IO connectors (e.g. USB ports, processor video (if available), keyboard, mouse, and 1 GbE network), power button and indicator light, two SATA connectors (as eSATA connectors), and a 10 GbE network connector.

===Blade Removal/Insertion===
On Limulus systems, blade removal begins by using the system control GUI (or command line tools) to remove power to the blade. Although blades are hot-pluggable, prior power removal ensures best results. Next the front of the case is removed to expose the blades (no tools needed). Once the blades are exposed, all network and data storage cables are easily removed. Finally, the thumb screws are loosened and used to pull the blade out. To replace the blade, the steps are performed in reverse. 

{{ :wiki:figure-4-node-power_ctl.png?400 |}}

**Figure Four:** //Power control applet for blade/node power control//

===Blade Power===
As mentioned, power is supplied though a single blind 12V (DC) power connector. Using an adaptor, this power rail is converted (efficient DC to DC) to the appropriate voltages required by the motherboard (see Figure Five). Also see [[Powering Up/Down Nodes|Powering Up/Down Nodes]] and [[Node Power Control|Node Power Control]].

===Video, Keyboard, Mouse, and USB Connections===
All motherboard ports are available on the front of the blade. In general, access to these ports is normally not needed when the system is operating as a unified cluster. If these ports are needed and if the blade processor provides an onboard video option (mainly Intel processors), then these ports can be used for debugging (or updating BIOS).  All blade ports are easily accessible by removing the front panel (no tools needed).

===Cabling===
All cables are secured to the blade to ensure nothing gets snagged or comes loose during operation.

===Blade Size===
Blades measure 11" x 11.3" x 1.9" (280mm x 287mm x 49mm), as shown in Figure Five.

{{ :wiki:figure-5-blade-top-annotated-size.png?400 |}}
**Figure Five:** //μATX blade dimensions with front and top view. Note: power connector is in lower right corner.// 

===Blade Enclosures===
A blade enclosure is needed to hold the blades in place. The enclosure must provide three features:

  - Blade rail slots to match the rails on the blades
  - An anchored nut receptacle for the thumb screws
  -A mate for the power connector

The following CAD drawing is for the enclosure that mounts in the standard Limulus case. Additional enclosure designs allow blades to be mounted in rack-mount cases. 

{{ :wiki:figure-6-limulus-blade-cage-front-side.png?400 |}}

**Figure Six:** //CAD drawings for Limulus desk-side case blade enclosure. The three blue cubes on the right side of the right image are the pluggable power receptacles.// 

===Reliability, Availability and Serviceability (RAS)===
As described above, blades are easily removed by users for return-to-depot service. Because all blades are powered independently, any blade can be removed and reinserted without affecting other operating (powered) blades. In addition, if on-site component repair is an option, the open design of the blade allows for fast and easy repairs using no more than a Phillips screwdriver. 

==== 3-D Printed Flexibility====
The use of 3-D printing helps lower construction cost and provides a new level of system flexibility.  Costs are lower because there is no need to buy sheet metal or injection-molded parts in high quantity. Since 3-D printed parts are printed "on demand," design changes and customized solutions are easily managed. In addition, 3-D printing can be used to create sophisticated shapes that might otherwise be expensive, difficult, or impossible to create with other manufacturing methodologies. 

====Environmental====
Limulus systems are designed to work in non-data center environments (Edge Computing). The Limulus 2.0 μATX blades are designed to run quietly (equipped with efficient CPU coolers and fans) and produce low amounts of heat due to the use of lower-TDP processors. 

Blades are designed to be used in close quarters in ventilated, yet rugged, environments. Other than the processor cooler, there are no moving parts. All non-blade storage is external to the blade, allowing multiple removable storage devices to be used with each blade.  All blades have at least one internal SSD NVMe slot.

A central issue for all computing equipment is the accumulation of dust within the components (due to the need for moving air to cool hot components). Because the blades are designed to use cooler processors, the amount of cooling (moving air) is less than for hotter processors. Thus, using a case with proper air filters (such as those used for Limulus systems), blades normally do not experience large amounts of dust accumulation. In addition, should dust find its way to a blade, it is easily removed and cleaned using compressed air. 

Concerns about 3-D printed parts in a "hot" computing environment are not an issue. First, systems are designed to run cool and have plenty of moving air and case ventilation to remove heat. Second, during normal operation, the hottest parts of the system are the CPUs and active coolers. These devices are not near or in contact with any 3-D printed parts. Modern processors can reach temperatures of 70-85 °C before they internally throttle the clock (thus reducing the amount of heat generated). Limulus nodes are also monitored and have a default throttling temperature (72 °C) that will initiate kernel-level throttling, if needed. Finally, all parts are printed (and annealed) using high performance PLA. The resulting parts start to become soft between 75-85 °C. It would take some work to seal off the case so that the internal temperature reaches this range. In addition, at these temperatures, many  other things would not work. 

====Conclusion====
The Limulus 2.0 μATX blade design provides convenient "big computing" to non-data center environments. Optimal design and the use of 3-D printing provides a powerful yet flexible approach to local computing solutions (e.g. for the office, lab, classroom, or factory). In addition, the use of high-quality commodity-priced components allows industry-leading price to performance. The modular design allows virtually any standard μATX motherboard and off-the-shelf hardware to be combined into a dense, high-performance computing resource.
Systems Manual

User Tools

Site Tools

Page Tools
