Zilliz announced that his latest research on vector databases was presented at the 48th International Conference on Very Large Databases (VLDB 2022). The research paper, titled “Manu: A Cloud-Native Vector Database Management System,” explains Manu, the project name for Milvus 2.0, which is an open-source vector database created by Zilliz. Xiaomeng Yi, co-author and senior researcher at Zilliz, presented the paper at the conference during an online session on managing specialized and domain-specific data.
Zilliz is one of the main contributors to Milvus, the vector database which the company says is specifically designed to process large-scale vector data to create AI applications for use cases such as computer vision, NLP, personalized research and the discovery of new drugs. Milvus features are integrated into Zilliz Cloud, a fully managed vector database service now available as a private preview.
Zilliz says Manu, or Milvus 2.0, is inspired by the research team’s interaction with more than 1,200 industrial Milvus users over the past three years. The authors claim that the main contribution of their VLDB paper lies in introducing the actual demand for vector databases and designing the basic architecture of a cloud-native vector database. With the development of learning-based integration models, some vector collections now number in the billions. “Manu relaxes data model and consistency constraints in exchange for the elasticity and scalability needed for a fully managed and horizontally scalable vector database. Manu is also extensively optimized for performance and usability with implementations compatible with hardware and support for complex search semantics,” the company said in a statement.
In a blog post, Principal Researcher Yi discusses how Manu, the second major release of Milvus, was designed with specific business needs in mind, including ever-changing requirements, more flexible consistency policy, component-level elasticity and a simpler and more efficient transaction processing model. Gathered from user feedback from the first iteration of Milvus, these requirements were combined with the more common requirements of a distributed system, leading to what Yi calls five big goals for Manu: long-term scalability, tunable consistency, good elasticity, high availability and high performance.
The research paper discusses how Manu supports these goals through its four-layer architecture, which it claims enables the decoupling of read from write, stateless state, and storage from computing. There is an access layer, a coordinator layer, a worker layer, and a storage layer, and a logging system as the backbone to connect the decoupled components. Yi explains that this log backbone, a core feature, enables independent scaling and evolution of each component and facilitates resource allocation and fault isolation.
Another main feature of Manu is its adoption of a delta consistency model which allows for adjustable levels of consistency. “Delta consistency ensures that updated data (including inserted and deleted data) can be queried and searched up to delta time units after the data update request is received by Manu,” said writes Yi.
The VLDB review board found that Manu/Milvus 2.0 “is one of the few specialized cloud-native vector databases available on the market” and that the team introduced “an efficient similarity search system in which the data entities consist of attributes and embedded vectors generated by machine learning models.
“Zilliz has been investing in frontier research since day one,” said Charles Xie, Founder and CEO of Zilliz. “Our dedicated R&D department helps us apply the latest Milvus research to continuously create value for our users. We are pleased that our article has been accepted by VLDB ’22, as the industry has once again recognized our technological leadership. We will continue to seek breakthroughs and apply the latest research results on vector databases in AI, in collaboration with institutions, communities and ecological partners.
To read more technical details about Manu, read Yi’s blog here.
Vector database company Zilliz raises $60 million to improve its cloud offering
Zilliz announces key contributions to Milvus 2.1
Home Depot finds DIY success with vector research