US and European companies tend to get all the attention when it comes to running open source in mega-scale production environments, but that’s a serious blind spot. It turns out that many Chinese companies are creating extraordinary, gargantuan-scale open source architectures, yet we know almost nothing about their innovations.

Yes, there are examples like the world’s fourth largest telco, China Unicom, which broadly embraced open source software to expand services and improve performance for its more than 320 million subscribers as the company rolled out its new 5G network ahead of the rest of the world.

Another interesting example is KE Holdings, also known as Beike Zhaofang, which debuted as a public company in August 2020. KE Holdings is a Chinese online real estate broker much like Zillow in the US. Unlike Zillow, KE Holdings has a market capitalization of $80 billion (four times more than Zillow’s). The heart of its service is built with open source.

Graph databases hit overdrive in China

With more than two billion real estate listings, KE Holdings’ cloud search architects struggled to keep up with the explosive growth, with added pressure from its imminent summer 2020 IPO. To improve performance of search results against the backdrop of massive user demand, the architects built a new stack on open source software that not only improved performance but also security while delivering more results faster and at a lower cost.

SEE: Special report: Prepare for serverless computing (free PDF) (TechRepublic)

“Open source is very important to KE for two main reasons: Costs and security,” said Pan Gao, chief search architect of KE Holdings. While Gao manages a team of 20 developers, the company overall employs more than 4,000 software engineers. His team’s stack is largely comprised of open source software, including Docker, Elasticsearch, Apache Hadoop, Apache Kafka, Kubernetes, Linux, MongoDB, MySQL, and Spark. But the beating heart of his high-speed search is a graph database from Dgraph Labs.

“We’ve shown that you can build a knowledge graph with 10 billion nodes and deliver millisecond query results and meet production environment requirements for SLAs and security,” Gao said. “We’re able to store 48 billion ordered triple datasets in Dgraph.”

I’ve written about Dgraph previously and how it’s helping to drive developer adoption of GraphQL, as well as the company’s recent launch of its Software as a Service (SaaS) offering called Slash GraphQL, which is designed to make it easy for developers to try GraphQL. Running GraphQL on a native graph database like Dgraph brings many benefits, but Dgraph can also be run standalone, as KE Holdings does.

The founder and CEO of Dgraph, Manish Jain, was a longtime software engineer at Google, pushing adoption of graph technologies across many of the search giant’s most popular products. After he left Google, Jain wrote a new graph database from scratch in Go and adopted GraphQL as its native query language in 2015 right after Facebook open sourced the technology. Then he founded Dgraph Labs, which today boasts customers such as American Express, Comcast, Intel, Intuit, Siemens, VMware, and more.

As big as graph databases are in the US, they took off much faster in China, according to Gao, starting about two years ago. On the other hand, GraphQL is only just now appearing on developer radars in China, whereas in the US, it’s taking off like a rocket.

Why the real estate giant chose Dgraph

So why did KE select Dgraph? Gao said his team first tried and failed with MySQL. Realizing that only a graph database could meet their needs, they tested other graph databases and ended up with JanusGraph and Dgraph as the final candidates. Dgraph won because it was open source and, according to Gao, only Dgraph was able to handle the volume of data KE had to manage and still deliver query results in milliseconds.

SEE: How to build a successful developer career (free PDF) (TechRepublic)

“The Dgraph database architecture is straightforward,” Gao said. “All of its functions are supported natively, so Dgraph does not depend on any third-party systems. It has only one executable file. By specifying the parameters on different local machines, it can automatically form clusters.”

Performance matters a lot, too. In performance comparison tests using a 30Gb data set, KE created an environment with three machines with 48 cores, 128Gb memory, and SATA hard drives. KE used 48 million data nodes, 63 million data edges, and 450 million triple data sets. Write performance was divided into real-time and pre-defined writes.

For nodal write performance, Dgraph reached 35,000/second, twice as fast as JanusGraph. For edge write performance, Dgraph was about 10% faster. Dgraph excelled in query performance. While it took JanusGraph 700 milliseconds to query third-degree node vertices and attributes, Dgraph only took a few milliseconds.

In charge of the cloud search platform for KE Holdings, Gao said he’s looking to next build a platform that integrates all the underlying layers of their current architecture: Effects platform, algorithm platform, the three retrieval engines (Elasticsearch, Dgraph, Faiss), and the container platform. He thinks Dgraph can support them all.

“In the future, the business side does not need to care about the underlying database storage, writing, and querying,” he said. “The new search platform will unify the relevant capabilities and provide a unified entrance and exit while ensuring the overall performance, stability, and security required to power our business. This will allow the business side to focus more on high-level strategic planning and business logic.”

Disclosure: I work for AWS, but the views expressed herein are mine alone.

Data, Analytics and AI Newsletter

Learn the latest news and best practices about data science, big data analytics, and artificial intelligence. Delivered Mondays

Sign up today

Also see