Software-Defined Storage 2.0: Do not break or stand, speed up the run

Mark Anderson said that “software is eating the world.” This is not alarmist. After entering the cloud era, everything seems to be happening. The original virtualization technology realized the abstraction of hardware details, making the infrastructure a single entity managed by software. The emergence of cloud computing has further turned hardware into an abstract resource, and cloud applications have turned hardware operation and maintenance into a service. The presence of hardware seems to be gradually declining in the cloud era. But in fact, with the development of cloud computing, a large number of opportunities for hardware innovation are also being created. For example, the emergence of software definitions has actually released more value and possibilities of hardware. Take software-defined storage as an example. Driven by many factors, it is also facing the status quo of being invincible. From the past 1.0 era to the new 2.0 era, speeding up has become the general trend. 1 Revisiting software definition Because of Tesla’s halo effect, “software definition”, a key word full of industry attributes, has entered the public’s field of vision. Tesla used a set of software-defined logic to change the competitive landscape in the automotive industry over the past century, and thereby promote software-defined cars, software-defined values, and even software-defined worlds. Back to the data center field, software definition originated from SDN (Software Defined Networking), and its concept originated in the Clean Slate research project of Stanford University in 2006. The software-defined data center was proposed by VMware in August 2012, and as a result, two new fields of software-defined storage (SDS, Software Defined Storage) and software-defined computing (SDC, Software Defined Compute) were created. In 2013, Gartner released the “software defines everything” among the top ten strategic technologies of 2014. Gartner believes that software defines everything including the growing market momentum under the improvement of infrastructure programmability standards, data center interoperability driven by the inherent automation of cloud computing, DevOps, and rapid infrastructure provision. ExponTech (Huarui Index Cloud Technology) CTO Cao Yuzhong said, “The three branches of software-defined computing, network, and storage have been recognized by the market in the past ten years. The essential development logic is that more and more capabilities are in software. The difference in hardware is shielded by software capabilities. For example, the emergence of the trend of cloud computing is the greatest achievement of software definition.” ExponTech CTO Cao Yuzhong Indeed, cloud computing uses a large-scale distributed software architecture to define the cloud infrastructure, and the underlying hardware is becoming more and more standardized. Taking the public cloud as an example, it emphasizes both the standardization of hardware and the abundance of upper-layer capabilities. The key lies in the software-defined layer. Therefore, the software definition itself can be seen as the cloud technology architecture model and the trend of technology development. Of course, Cao Yuzhong also pointed out, “In the process of software definition, software definition does not determine everything. We are also seeing new trends. For example, when business scenarios and performance have higher requirements, some new hardware will also appear. The form requires the cooperation of software and hardware. For example, in AI scenarios, software definition is required to adapt to new hardware.” From the perspective of software-defined storage, in fact, the relationship between software and hardware needs to be re-examined. From the customer’s point of view, software and hardware must be decoupled and should not be bound to specific hardware. In terms of product form, it may be a combination of software and hardware, or it may be pure software. Therefore, we see that the delivery model of software-defined storage includes both pure software and an integrated delivery form of software and hardware. There is no absolute boundary between them and ultimately depends on the actual service needs of customers. 2 Software-defined storage space and layout From the perspective of the software-defined data center, the development of software-defined storage is gaining momentum. Data show that the overall software-defined storage market will exceed 10 billion yuan in 2020, accounting for nearly 30% of the enterprise-level storage market. Globally, some analysts predict that data shows that by 2023, the global SDS market will exceed 80 billion US dollars. With such a huge development space for software-defined storage, Cao Yuzhong analyzed the key points from both macro and micro dimensions. #01 From a macro perspective First of all, the development of the software-defined storage market and the cloud computing market are complementary. Based on software definition, to build a storage resource pool, which is consistent with the concept of cloud computing, software definition and cloud are a perfect match. Therefore, the bottom layer of cloud computing tends to be distributed storage technology. As all industries are moving towards cloudification, software-defined storage will be more developed, and its development trend will be synchronized with the development of cloud computing. Secondly, it is social demand. Human society is entering the digital society at a fast pace, and technologies driven by digital technologies such as the Internet of Things and AI are constantly erupting. These technologies are based on massive data operations, and propose flexibility and scalability for data storage and processing. higher requirement. Compared with traditional storage, software-defined storage ensures that under the premise of a user-controllable TCO (total cost of ownership), it also has the characteristics of agility and efficiency, on-demand deployment, stability and reliability, and is especially suitable for dealing with the challenge of massive data. #02 From a micro perspective First of all, the digital transformation of all industries in the future, and industries where enterprise IT cloud transformation is prosperous, will usually choose software-defined storage technology. Because they need to meet the requirements of business innovation through rapid cloud transformation. Fast-transforming industries such as new consumption and the Internet are the markets with the most intensive demand for software-defined storage applications. Secondly, among the traditional industries where digital transformation is relatively fast, especially industries that need to process massive amounts of data, such as autonomous driving in the automotive industry, according to Intel’s survey data, each driverless vehicle will generate about 4TB of data per day… They are also one of the main driving forces for the software-defined storage market to accelerate. 3 Out of open source technology, architectural innovation From 0 to 1 of any technology is the most difficult process, and the application of open source technology has contributed to the development of software-defined storage. The most typical one is Ceph. Ceph’s unified and distributed design concept realizes a set of storage that provides multiple storage services for objects, blocks, and files at the same time. It not only meets the needs of different applications, it greatly simplifies storage deployment and operation and maintenance, but also allows the scalability and scale of the storage system. It is no longer restricted, thus promoting the development of software-defined storage. But open source does not mean everything. It is more like a double-edged sword. It is necessary to objectively look at the value of open source technology to the future of software-defined storage. On the one hand, open source technology has important value in rapidly popularizing technology and expanding the ecosystem. For example, Ceph has played an important role in the field of software definition, cultivating talents, lowering technical thresholds, and promoting the widespread use of technology. On the other hand, open source technology also has its disadvantages. Many open source projects focus on new features and technical highlights, and it takes less time to polish the product, resulting in overall deliverability, operation and maintenance. The ease of use and other non-functional features are weak or even missing, and it is difficult for ordinary small and medium-sized teams to have the energy and sufficient technical capabilities to complete the commercialization of open source technologies. Cao Yuzhong also reminded, “To make good use of open source, a strong team is needed, which means high costs for enterprises.” This is also the reality of many software-defined storage projects. Some open source projects have focused on small improvements in the past ten years, lacking fundamental changes. If no changes are made at the architectural level, software-defined storage will face more thorny challenges in the next few years. The 2.0 era of software-defined storage came out at the historic moment. 4 Software-defined storage Era change from 1.0 to 2.0 We know that in any era, there must be important driving factors. Software-defined storage 1.0 has gone through more than ten years, and now it is about to enter the 2.0 era, what are its driving factors? First, look at the industry perspective. The digital transformation of the industry has given birth to a large number of popular emerging technologies. Faced with the demand for massive data processing, data storage is required to be able to support the development of emerging technologies. The architecture of software-defined storage 1.0 was born before 2010, and it is difficult to foresee today’s technology and market demand. Second, look at the hardware development perspective. Ten years ago, the mainstream hardware was mechanical hard drives with low-speed networks of around 1G. Today, SSD solid-state hard drives are gradually replacing mechanical hard drives, and the network has also been upgraded from 100M and Gigabit to 100G or even 400G. This also requires a new software-defined storage architecture to meet the needs of hardware iteration. Third, look at the perspective of cloud computing architecture. Software-defined storage 1.0 is mainly to build a virtualized environment in the customer’s data center. Nowadays, the development trend of distributed cloud has made the storage architecture in public clouds, even multiple public clouds, as well as core data centers and edge data centers, and the cloud computing environment has become complicated. This also requires software-defined storage architecture upgrades to adapt to the new The change. Fourth, look at the data perspective. The software-defined storage 1.0 architecture was born out of storage thinking, based on the dimension that data is a derivative and accessory of an application. But now data is the driving force of the application, and the application is to play the value of the data. Software-defined storage architecture needs to be upgraded to a new data-centric perspective. It is precisely because of these key factors that ExponTech (Huarui Index Cloud Technology) put forward the concept of software-defined storage 2.0. We can indeed see the new value of software-defined storage 2.0 from the overall product logic. #01 Support end-to-end composable architecture From the perspective of data perception and IO perception, it combines and integrates storage resources in distributed cloud environments (private clouds, multiple public clouds, edge), providing different IO models, different performance and reliability requirements, and many types of protocol interfaces (iSCSI, S3) , POSIX, NFS, CIFS, CSI, HDFS, etc.) can be freely combined and deployed flexibly. Scale-up and Scale-out can be expanded arbitrarily. One data platform and one data support diversified business loads; #02 Architecture adapts to distributed cloud environment A set of software stack covers multiple scenarios of public cloud/private cloud/edge cloud, which can span data centers, multiple public clouds and edges, to achieve full transparent management and scheduling of data, business perception and data flow from a business perspective. The storage location tends to be transparent, and any location business can access any location data; #03 Architecture and cloud native environment adaptation Full-containerized deployment and micro-service management to achieve minimal deployment across hybrid multi-clouds and edges. Data storage/services can be orchestrated and agilely distributed anywhere in the cloud and edge, and data flow can be self-adapted and strategically scheduled to improve Adapt and support cloud native applications; #04 A new generation of distributed storage engine Realize complete support for new storage media and emerging technologies, including adapting new media or new technologies such as SSD, SCM, NVMe-oF, RoCE, etc., and give full play to the capabilities of these new technologies to adapt to massive data storage and analysis and production core Business requirements; #05 Have a more comprehensive data perspective Rather than just a storage perspective, the overall consideration of southbound data storage and northbound data services can seamlessly evolve to a data lake, a lake warehouse integrated platform, and a hybrid multi-cloud data platform by means of combined and superimposed service components. It is not difficult to find that the software-defined storage 2.0 defined by ExponTech is not completely supported by ideas, but also has support for solutions that can be implemented and a sufficient set of capabilities. 5 postscript Cao Yuzhong said, “We hope to surpass the limitations of SDS only focusing on the storage layer, and use a new generation of distributed architecture technology to help enterprises build a data-centric architecture, better adapt to the trend of distributed cloud, and realize a data platform. , To store and manage all the data in the enterprise.” It is based on this vision that ExponTech has planned a clear product goal and development path. Based on a combined architecture, a unified storage engine, a unified metadata service engine, etc., different service components are introduced rhythmically, including various storage coordination interfaces and various data-related services. Regarding the connection of the past software-defined storage 1.0, Cao Yuzhong said frankly that ExpoTech will also give priority to compatibility with the original open source ecosystem, and seamlessly connect the original Ceph-based customers. At the same time, it will aggregate industry partners to build SDS2.0. The new ecosystem of the times. All great companies are derived from a clear vision. ExponTech, which defines the era of software-defined storage 2.0, is striding forward on this road