busy hyperscaler bees buy hardware, not software – Blocks and Files


Hyve Solutions sells storage hardware to its hyperscaler customers, but not software, because they don’t want it. So what do hyperscalers want?

Hyve is a business unit of TD Synnex that sells, delivers and deploys computing infrastructure (compute, network and storage) to hyperscaling customers. That means the likes of AWS, Azure, Facebook, Twitter, etc., although that would only confirm Facebook as a customer. Typically the hardware is rack scale and conforms to Open Compute Project standards. Hyve also sells IoT and edge deployments, such as those using 5G, where under-rack systems can be typical and site numbers can number in the hundreds.

Jay Shenoy.

We were briefed by Jayarama (Jay) Shenoy, Hyve’s VP of Technology, and our conversation focused on drive and rack connection protocols to SSD form factors.

Shenoy said Hyve is an original design manufacturer (ODM) and focuses exclusively on customers and hyperscaler hardware. It doesn’t really have separate storage products – SAN and filers for example – with their different software. Separated, i.e. servers.

Shenoy said, “We build boxes, often to the requirements and specifications of our hyperscale customers. And we have very little to do with the software part.

Storage and storage servers

Initially, hyperscalers used generic servers for everything. “A storage server differed from a compute server only in the number of disks or SSDs. … There really wasn’t much of a difference at times on the hardware, being the storage. That changes now. But the change will show up in, I would say, two or three years. Things are moving much slower than we would like. »

One of the first big changes to the storage area was a change in drive type. “So the first thing that happened was that HDDs gave way to SSDs. And initially, for several years, SSDs were a test. Now that trend is really, really entrenched , so you can just assume that the standard storage device is an SSD.

As a result, there has been a change in the player connection protocol. “Over the past three years, SATA has started to give way to NVMe, to the point where now SATA SSD would be a novelty. … We still have a legacy, things that we continue to ship. But in the newer designs, it’s almost all NVMe.

Shenoy switched to player formats. “Three years ago, hyperscalers were sort of split between U.2 [2.5-inch] and M.2 [gumstick card format]. All the M.2 folks kind of had buyer’s regret. … All three have confirmed that they have migrated or are migrating to the E1.S form factor.

E1.S is the M.2 replacement design in the Enterprise and Datacentre SSD Form Factor (EDSFF) ruleset for newer SSD form factors.

Shenoy said, “There are mainly three, thank goodness. The whole form factor discussion has been settled. This happened maybe a year and a half ago when Samsung finally ditched its NF1 and switched to EDSFF.

There are two basic EDSFF formats: E1 and E3. Each comes in short and long sub-formats, and also with varying thicknesses:

Details of SNIA’s EDSFF form factor.

The E3 form factor allows for a PCIe x4, x8, or x16 host interface. Shenoy’s basic trio is E1.S, E1.L, and E3.

Shenoy said Blocks and files “At a high enough level, what seems to me to be happening is that E3 is stepping into… what would have been known. What’s still known as enterprise storage with features like dual porting….E1 doesn’t really support dual porting.

Hyperscalers have adopted the E1 format, “but to date, I have not come across a single one that has an E3 format [drive].” He explained that “people who chose U.2 are satisfied with U.2 and will like U.3 instead of E3”.

The difference for hyperscalers between E1.S and E1.l format is capacity. “[With] E1.S compared to E1.L, the difference is exactly one capacity point, which means the highest capacity point in the SSD lineup. … Only the port with the highest capacity will be offered as E1.L. And everything else will be offered in E1.S. So even .L is essentially limited to anyone who will adopt the 30 terabyte disk capacity point today.

Advanced Computing

“In terms of hardware, there’s another subtle, subtle – well, actually, not so subtle – change that’s happened at hyperscale. When we think of hyperscale, we think of large data centers , to the hundreds of thousands of nodes, and to the racks, and the racks, and the racks, and the racks. But edge computing…stopped being a buzzword and started to be a reality for us at over the past two years.

“The way edge computing is changing storage requirements is that SSD hot swapping has returned. Hotswap – or at least easy field maintenance – has returned as an absolute requirement. Shenoy thinks “Edge computing driving different server form factors is a no-brainer.”

Another change is disaggregation. Shenoy said, “The biggest hyperscalers have basically embraced network attached storage. So it’s basically object storage or object-like storage for petabytes or exabytes of data [and] which is placed on hard drives. And SSD is fast storage. It is a combination of both: it can be object storage or sometimes even block storage. »

PCIe account

What about connection protocols in the hyperscaler world of Hyve? For them, “outside the rack…can do anything as long as it’s Ethernet. In the rack, there’s a lot more leeway as to what cables people will tolerate and what exceptions people are willing to make. Think of hyperscale storage chassis as essentially flash memory or disk enclosures – JBOF or JBOD. Most often they will be connected with a large PCIe cable. According to Shenoy, “PCIe in a rack and Ethernet outside a rack, I think that’s kind of a thing or a reality already.”

What about PCIe generation transitions – Gen 3 to Gen 4 and Gen 4 to Gen 5? Shenoy said there are still very few Gen 4 endpoints, especially GPUs, which “were almost ready before server processors. … Part of that is AMD‘s success, because Rome had to be the first with PCIe 4.” In fact, the transition to “PCIe gen 4” happened quite quickly. As soon as the server processors were there, at least some of the endpoints were there. The SSDs took a bit of time to move to gen 4.”

But PCIe 5 is different: “Gen 5…. and the protocol that writes on top of Gen 5, CXL, turns out to be…very different. It turns out to be like the transition from Gen 2 to Gen 3, where the processor appeared and no one else did.

According to Shenoy “Gen 5 network adapters are also lacking…to the point that the first processor to carry Gen 5 will probably go, I don’t know, a good part of a year without actually connecting to Gen 5 devices. . It’s a long time.”

Shenoy held on to this point. “The gen 5 transition is a bit like the gen 3 transition. … Gen 3 has been sticking around for a long, long time.

Chassis and rack

Hyve has standard server and storage chassis building blocks. “We have our standard building blocks. And then we have three levels of customization. …So the chassis is one of the things that we customize quite frequently. Edge computing systems vary from rack to sub-rack. Edge IT racks can be more sparse (Shenoy’s term) than data center racks. Each, without exception, will contain individual servers and may also contain routers and even long distance connection devices.

They look more like converged systems than normal Hyve data center racks.

Number of clients

How has the number of Hyve hyperscaler customers changed over the years? Shenoy said, “The number of hyperscalers has increased, or the number of Hyve customers has increased slightly, I would say, over the past three years.

“The central hyperscale market has not changed. Three or five years ago you had the big four in the United States – the big five if you count Apple – and then the big three in China. And then there was a bunch of what Intel calls the next wave” – that means companies like eBay and Twitter. “These would have been known ten years ago as CDN. Today it would be a kind of edge computing. I refer to them without [saying] whether they are our customers or not.

He said: “Then the other kind of customers that came to us…are the 5G telecom operators.”

This means that the addressable market for Hyve in terms of number of customers three to five years ago was around ten, and now it’s bigger than that – perhaps in the range of 15 to 20. C t is a specialized market, but its purchasing power and design influence – witness OCP – is immense.

This design influence affects drive form factors, as evidenced by the EDSFF standards set to replace the M.2 and 2.5-inch SSD form factors. Nearline high-capacity disk drives will remain powered on in their old 3.5-inch drive bays. The primary development goal of EDSFF was to get more SSD capacity into the server chassis while overcoming the power and cooling requirements that come with it.

Consumer enterprise storage buyers probably won’t come into contact with Hyve unless they start rolling out large numbers of edge sites using 5G connectivity. Other than that, Hyve is expected to remain a specialist provider of hyperscale IT infrastructure.

Boot note: Distributor Synnex got involved in all of this when it started providing Facebook’s IT infrastructure. This prompted him to start his company Hyve ten years ago. The rise of OCP and its adoption by hyperscalers has propelled Hyve’s business to the top.

The TD part of the name comes from Synnex’s merger with Tech Data in September last year, with combined annual revenues reaching nearly $60 billion. This has made it the largest distributor in the computer industry.


Comments are closed.