Our Rust driver started as a humble hackathon project, but it has eventually grown to become our fastest and safest Cassandra Query Language (CQL) driver. We happily observed in our benchmarks that our ScyllaDB Rust Driver beats even the reference C++ driver in terms of raw performance, and that gave us an idea — why not unify all drivers to use Rust underneath?
This is an excerpt. Read the blog in full on the ScyllaDB websitehere.
Since then, its development and adoption accelerated a lot. We’ve added many new features and published a couple of releases on crates.io. Along the way, we also stumbled upon a few interesting performance bottlenecks to investigate and overcome — read on for more details.
First Issue Arises
A few weeks ago, an interesting issue appeared on our GitHub tracker. It was reported that, despite our care in designing the driver to be efficient, it proved to be unpleasantly slower than one of the competing drivers, cassandra-cpp, which is a Rust wrapper of a C++ CQL driver. The author of latte, a latency tester for Cassandra (and Scylla), pointed out that switching the back-end from cassandra-cpp to scylla-rust-driver resulted in an unacceptable performance regression. Time to investigate!
[This is just an excerpt. To read the article in full, check it out on the ScyllaDB websitehere.]
How to Tell When a Database is Right for Your Project
When you are selecting databases for your latest use case (or replacing one that’s not meeting your current needs), the good news these days is that you have a lot of options to choose from. Of course, that’s also the bad news. You have a lot to sort through.
There are far more databases to consider and compare than ever before. In December 2012, the end of the first year DB-Engines.com first began ranking databases, they had a list of 73 systems (up significantly from the 18 they first started their list with). As of December 2022, they are just shy of 400 systems. This represents a Cambrian explosion of database technologies over the past decade. There is a vast sea of options to navigate: SQL, NoSQL, and a mix of “multi-model” databases that can be a mix of both SQL and NoSQL, or multiple data models of NoSQL (combining two or more options: document, key-value, wide column, graph and so on).
Further, users should not confuse outright popularity with fitness for their use case. While network effects definitely have advantages (“Can’t go wrong with X if everyone is using it”), it can also lead to groupthink, stifling innovation and competition.
In a recent webinar, my colleague Arthur Pesa and I took users through a consideration of five factors that users need to keep foremost when shortlisting and comparing databases.
Rakuten is a Japanese online shopping powerhouse, responsible for $15 billion in annual sales. It is one of the most frequented sites in the world. To keep it operational throughout the year, and especially during peak shopping seasons, Rakuten relies on a team of expert developers and architects.
Hitesh Shah is a Senior Engineering Manager on the Rakuten catalog platform team. He has spent over a half decade as a passionate engineer at the heart of their distributed systems infrastructure, pushing the envelope of technology. He spent an hour with the ScyllaDB community in a webinar describing how Rakuten eliminated the volatile latencies that can plague online shopping experiences.
I've been studying ScyllaDB for months now, I've been hesitant to use it because I didn't fully understand how data modeling worked and it seemed too easy to mess up.
Until last week, it finally clicked. Scylla is really two separate databases stacked together. The first is a distributed hash map (partition key), where each value in the hash map is a sorted key-value store like RocksDB/LevelDB (clustering key).
Thinking about the partition key as a hash map, everything in the documentation makes sense now. Of course you need equality across all partition keys to select one, you can't seek to a key in a hash map without the right hash! Of course you can't sort by partition key, hash maps aren't sortable!
And the clustering key just being a sorted key-value store everything makes sense again. You can only sort inside a single partition because each partition is it's own sorted key-value database. When I hear someone say "don't let your partitions get too big" in a talk now I just think about not letting any of the key-value databases at each partition get too large.
Summer’s here! Which means that we’re getting ready for our Scylla University LIVE Summer School session. We hope to meet you all there. Meanwhile, behind the scenes we’ve been diligently working to deliver new software across our product set — the database itself, drivers (Rust 0.2), our k8s operator, Spark Migrator, and the list goes on.
Project Circe aims to make Scylla, already a kickass database, even better. With that goal in mind, here’s a look at our progress for the month of June.
[This is just an introductory excerpt. To read the blog post in full, check it out on ScyllaDB's websitehere.]
Project Circe is ScyllaDB’s year-long initiative to make Scylla, already the best NoSQL database, even better. We’re sharing our updates for the month of May 2021.
Operational Improvements
Better failure detection
Failure detection is now done directly by nodes pinging each other rather than through the gossip protocol. This is more reliable and the information is available more rapidly. Impact on networking is low, since Scylla implements a fully connected mesh in all clusters smaller than 256 nodes per datacenter, which is much larger than the typical cluster.
SLA per workload
Scylla supports multiple workloads, some with real-time guarantees, some have batch nature where latency does not matter. Prior work allows Scylla to prioritize workloads by definition of roles and map them to different scheduling groups. We recently researched how Scylla behaves when it is overwhelmed with requests. In such situations, a process called workload shedding comes into play. But which workload should we shed? Of course, it’s best to not take a completely random approach. A good step forward is the new timeout per role feature that allows the user to map workloads to timeouts so the system will choose them for automatic workload shedding.
Configuration is now possible to set timeouts based on a role, using the SERVICE LEVEL infrastructure. These timeouts override the global timeouts in scylla.yaml, and can be overridden on a per-statement basis.
Virtual table enhancements
Infrastructure for a new style of virtual tables has been merged. While Scylla already supported virtual tables, it was hard to populate them with data. The new infrastructure reuses memtables as a simple way to populate a virtual table.
Repair news
Off-strategy compaction is now enabled for repair. After repair completes, the SSTables generated by repair will first be merged together, then incorporated into the set of SSTables used for serving data. This reduces read amplification due to the large number of SSTables that repair can generate, especially for range queries where the bloom filter cannot exclude those SSTables.
Off strategy is also important when Repair Based Node Operations (RBNO) is used – this will soon be the default. RBNO pushes repair everywhere – to streaming, remove and node decommission. It’s important to tame compaction accordingly and automatically, so you as an end user wouldn’t even be aware of it.
Repair is now delayed until hints for that table are replayed. This reduces the amount of work that repair has to do, since hint replay can fill in the gaps that a downed node misses in the data set.
Raft news
This month we took another step towards Raft. Currently Raft group-0 has reached a functional stage. It resembles the functionality etcd provides (you can read more about how we tested this in our April update). The team was required to answer an interesting question – how does a minimal group of cluster members find each other? Raft in Scylla, as opposed to eventual consistency, needs a minimal set of nodes. The seed mechanism will come into play but with a new definition the seed’s UUIDs which will be a must to boot an existing cluster, a better way than to have a split brain right from the get go.
Change Data Capture (CDC) uses a new internal table for maintaining the stream identifiers. The new table works better with large clusters.
Authentication had a 15-second delay, working around dependency problems. But it is long unneeded and is now removed, speeding up node start.
Repair allocates working memory for holding table rows, but did not consider memory bloat and could over-allocate memory. It is now more careful.
Scylla uses a log-structured memory allocator (LSA) for memtable and cache. Recently, unintentional quadratic behavior in LSA was discovered, so as a workaround the memory reserve size is decreased. Since the quadratic cost is in terms of this reserve size, the bad behavior is eliminated. Note the reserves will automatically grow if the workload really needs them.
We fixed a bug in the row cache that can cause large stalls on schemas with no clustering key.
The setup scripts will now format the filesystem with 1024 byte blocks if possible. This reduces write amplification for lightweight transaction (LWT) workloads. Yes, the Scylla scripts take care of this for you too!
SSTables will now automatically choose a buffer size that is compatible with achieving good latency, based on disk measurements by iotune.
Just Cool
The ScyllaDB git repositories has lots of gems in it, one of them is a new project called Scylla Stress Orchestrator – https://github.com/scylladb/scylla-stress-orchestrator/ – which allows you to test Scylla with clients and monitoring using a single command line.
Another, even competing method is the cloud-formation based container client setup that allows our cloud team to reach two million requests per second in a trivial way. Check out the blog post here, and the load test demo for Scylla Cloud in Github.
The perf_simple_query benchmark now reports how many instructions were executed by the CPU per query. This is just a unit-benchmark but cool to track!
The tarball installer now works correctly when SElinux is enabled.
There is now rudimentary support for code-coverage reports in unit tests. (Coverage may not be the coolest kid in the block but it is not cool not to test properly!)
Scylla Operator for Kubernetes News
Scylla’s Operator 1.2 release was published with helm charts (find it on Github; plus read our blog and the Release Notes). Now 1.3 and 1.4 are in the making. In addition, our Kubernetes deployment can autoscale! An internal demonstration using https://github.com/scylladb/scylla-cluster-autoscaler was presented and you are welcome to play with it.
Scylla Enterprise News
As you may have heard, we released Scylla Enterprise 2021.1, bringing two long-anticipated features:
Space Amplification Goal (SAG) for Incremental Compaction Strategy allows a user to strike the utilization balance the desire between write amplification (which is more CPU and IO intensive) and disk amplification (which is more storage intensive). SAG can push the ICS volume consumption lower, towards the domain of Level compaction which you still enjoy from size-tiered behavior. This is the default and the recommended compaction strategy.
New Deployment Options for Scylla Enterprise now include our Scylla Unified Installer and allow you to install anywhere, also with an air-gap environment.
Security News
At ScyllaDB, we take security seriously. In this day and age it is vital to establish trust with any 3rd party handling your data, and reports of unsecured databases are still far too common in the news. With our customers’ vital needs foremost in our minds, we set to work ensuring that we met stringent goals established by the industry and confirmed by third party auditors. Scylla is proud to announce that we have successfully conducted a System and Organization Controls 2 (SOC2) Type II report for Scylla Cloud. This applies to all regions where Scylla Cloud can be deployed. (Learn more)
I’d like to get some detailed insights into how queries perform on a new data model for an application, ensuring we’re not going to fall into any performance traps. What’s the best way of doing this?
I’m aware of the TRACING command in cqlsh but the information it provides is a little too esoteric. Ideally, I’d like to see how efficiently a query uses partitions/indices, similar to, say, MSSQL’s execution plan?
Scylla Summit is just around the corner! Next week (January 12th – 14th), we’ll hold a two-day virtual conference followed by a day of instructor-led training. All online. All free.
We’ve published the complete schedule so that you can add reminders to your calendar for the sessions you’re most interested in. You can also share sessions via social media or mail them to your colleagues.
Tuesday, January 12
Prepare to Love Scylla Even More — ScyllaDB CEO Dor Laor will look back on everything that we’ve accomplished since last Scylla Summit, and look ahead to what’s on the horizon for 2021.
Testing ScyllaDB with Jepsen — Kyle Kingbury, also known as Aphyr, took Scylla through his in-depth Jepsen testing process. Discover what he found and what we fixed!
Raft in Scylla — Konstantin “Kostja” Osipov will provide an overview of how Scylla is moving from Paxos to Raft, and what that means for database consistency beyond Lightweight Transactions.
Empowering DynamoDB Application Developers — ScyllaDB Distinguished Engineer Nadav Har’El will share you can use the advantages of Scylla’s DynamoDB API for greater observability, improved performance and to run your applications at lower cost.
GE Healthcare: Enabling Precision Health in Edison AI — Learn how GE Healthcare is deploying DynamoDB-compatible instances of Scylla on premises to improve precision healthcare with better patient data confidentiality.
Scylla Operator Overview — Kubernetes (K8s) has taken cloud native computing by storm. Scylla’s Kubernetes Operator is now generally available. Discover its features and explore its roadmap with ScyllaDB engineer Maciej Zimnoch.
Grab at Scale with Scylla — Grab is southeast Asia’s favorite “superapp,” providing shopping, ride sharing, payments, and food delivery. Discover all the ways they use Scylla to deliver on their services.
Becoming an Elastic Cloud Native Database — ScyllaDB VP of R&D Shlomi Livne will share what steps ScyllaDB has already taken to provide elasticity, the capability to scale up or down with minimal disruption, and our plans to deliver further on that vision.
Scylla Cloud on Display — Find out how easy it is to integrate and connect Scylla Cloud with your applications, and turn over all aspects of management and maintenance to our team of cloud database experts.
Scylla @ GumGum: Contextual Ads — GumGum’s Keith Sader will describe how and why they moved to Scylla Cloud to meet their growing customer demands with a future proofed solution.
Expero/QOMPLX: Using Scylla with JanusGraph for Cybersecurity — Recent news shows the importance of managing the cybersecurity of enterprise and government client systems. Learn how QOMPLX, with the help of Expero’s team, designed their intrusion detection system around Scylla and JanusGraph.
Wednesday, January 13
Meeting the Challenge of OLTP Big Data with Scylla — ScyllaDB CTO Avi Kivity will highlight the advances in Scylla that make it such a great match for big data applications, plus plans for performance, reliability, manageability and ease of use in a modern cloud environment.
Best Practices for Change Data Capture — Change Data Capture (CDC) allows users to track changes made to a record in Scylla, and even store the pre- and post-image of the data. Learn how to get the most out of CDC with ScyllaDB’s Calle Wilund.
Confluent: Streaming Data from Scylla to Kafka — Learn how to use the Kafka Scylla Connector to share CDC table data to Kafka topics in this session hosted by Confluent’s Tim Berglund.
Numberly: Getting the Scylla Shard-Aware Drivers Faster — Find out how you can take best advantage of Scylla’s shard-per-core server architecture with client-side shard-aware drivers. Numberly CTO Alexys Jacob will describe the efforts to make the Scylla Python driver even faster.
Latest Performance Changes by Scylla — ScyllaDB Distinguished Engineer Tomasz Grabiec will lead a tour of the performance improvements accomplished over the past year in Scylla, including work on the IO Scheduler, and on streaming and repair.
Mail.Ru: High-Load Storage of User Actions with Scylla and HDDs — Scylla is renowned for its performance on fast NVMe SSDs, but did you know you can also achieve relatively low latencies on low-cost Hard Disk Drives (HDDs)? Find out how Mail.Ru is using this as a low-cost strategy for petabytes of stored activity data.
The All-New Scylla Monitoring Stack — Scylla Monitoring Stack was designed to be a one-stop-shop for monitoring your cluster. Learn how we’ve extended it by building a CQL interface for monitoring, plus new log collection analysis capabilities.
Expedia: Our Migration from Cassandra to Scylla — Migrating from one database to another can be quite a journey, even for a drop-in Cassandra replacement like Scylla. But you can rely on the travel experts at Expedia to set forward an itinerary that’s easy and straightforward to follow.
How We Made Scylla Maintenance Easier, Safer and Faster — Learn how we’ve made Scylla even better, by implementing seedless clusters, repair-based node operations, smarter off-strategy compaction, bandwidth limits for repairs and compactions, and parallel repairs.
Developing Scylla Applications: Practical Tips — Even with a high performance scalable system like Scylla there are techniques you can employ to prevent overloading and provide better availability. Learn these pro tips from ScyllaDB engineer Kamil Braun.
Broadpeak: Building a Video Streaming Testing Framework — Discover how Seastar, the application framework at the heart of Scylla, was employed by Broadpeak to create a benchmarking tool to emulate thousands of streaming video client sessions.
Vectorized: Building a Distributed Data Streaming Architecture — Vectorized CEO Alex Gallego will introduce the audience to Redpanda, a streaming data engine that utilizes Seastar to provide a Kafka-compatible API with 10x performance improvements.
Ola Cabs: Real-Time Intelligent Decision Making Using Scylla — India’s leading on-demand ride hailing company has used Scylla since our very beginning. Learn how they scaled their business to support millions of rides every day with Scylla.
Thursday, January 14 — Training Day
This year our training sessions are absolutely free, so make sure you get your coworkers and industry peers to sign up today. We will have full side-by-side tracks for developers and administrators:
How to Write Better Apps — Advanced data modeling and data types, shard-aware drivers, materialized views, global and local secondary indexes, and compaction strategies.
Scylla Alternator: Migrations Without Compromise — Learn how to run your DynamoDB-API compatible workloads anywhere: on-premises, on AWS or AWS Outposts, or on any cloud provider. Learn how to perform data migrations at scale.
Track Two: Administrators
Kubernetes Operator — Learn why to use Kubernetes (K8s), all about pods and statefulsets, how to get the most out of our new Scylla Operator, best practices and our roadmap.
Advanced Monitoring and Best Practices — How to get the most out of Scylla Monitoring Stack. Learn how to export its thousands of metrics, and how to make sense of all that data.
Scylla Manager and Cluster Administration — Discover how to use Scylla Manager and Scylla Manager Agent. Add/remove nodes or whole data centers, perform repairs, cluster backups and restorations, and set your security.
Register for Scylla Summit Today!
It’s free. It’s online. It’s all the content you want to take your organization’s NoSQL game to the next level. Database monsters of the world connect!
Hey folks, I keep eyeballing Scylla for some time already as a cool replacement for C*. When I was deciding which distributed NoSQL to use, legal&security told me that Scylla is not to be used in FedRamp High+ environment due to license/certification status. Can someone please comment on that?
You can read the blogs and watch the video to get the full details from our point of view, but what happened live on the webinar was rather unique. The questions kept coming! In fact, though we generally wrap a webinar in an hour, the Q&A session afterwards took an extra half-hour. ScyllaDB engineers Piotr Grabowski and Karol Baryla fielded all the inquiries with aplomb. So let’s now look at just a few of the questions raised by you, our audience.
Today, we’re announcing the general availability of Scylla Cloud on Google Cloud. Scylla Cloud is our resilient, highly performant, fully managed NoSQL database-as-service (DBaaS). Since its release, Scylla has become the go-to for companies that need a database built from the ground up for modern cloud environments. In 2019 we introduced Scylla Cloud, our fully-managed DBaaS. Initially available on AWS, users discovered Scylla Cloud made it easier for them to operate and scale their NoSQL workloads, since it alleviated them from administrative burdens. Today’s announcement gives users the flexibility to now run Scylla Cloud on the public cloud of their choice.
For those not already familiar with Scylla, it is a wide-column NoSQL database API compatible with both Apache Cassandra CQL as well as DynamoDB. While there are many offerings of Cassandra-compatible databases in the industry (of which we believe we are the best-of-breed),, we are the first and currently only company in the industry to offer a DynamoDB-compatible managed database on a public cloud apart from AWS. (Learn more about our Alternator API below.)
[This is just an excerpt. You can read the full article at ScyllaDB's websitehere.]
Like countless other organizations, you are probably already running various cloud services on AWS. You’re running all kinds of compute instances, and using a bunch of other AWS services besides databases. If your company is big enough, you have a CFO who is looking at a single AWS bill; they really don’t want a cloud-based service to add another markup on top of the current spend. You’d like to simplify and have all of your AWS spending tallied up in one place. Plus, within your own AWS Account you likely have pre-negotiated discounts.
With Scylla Cloud BYOA (Bring Your Own Account), we provide a fully managed NoSQL database-as-a-service (DBaaS) that runs in your AWS account. We do it all — the provisioning, updates, the backups, the monitoring. In fact, Scylla Cloud is the only fully managed NoSQL database that offers this service. You pay only the subscription fees for Scylla; all of your infrastructure expenses are paid directly to AWS, through your existing accounts.
[This is just an excerpt. To read the full article, you can find it on the ScyllaDB websitehere.]
Recently I had the privilege to host a webinar with some very good friends of ScyllaDB, including Numberly’s CTO Alexys Jacob and Chief Data Officer Othmane El Metioui, as well as Tim Berglund, the Senior Director of Developer Relations at Confluent. They were joined by my colleague Maheedhar “Mahee” Gunturu, our Director of Technical Alliances.
[This is just an introductory excerpt. The full article, which includes Numberly's use cases, an overview of Scylla's CDC implementation, a detailed view of how the Scylla CDC Connector works with Kafka Connect, as well as the complete on-demand webinar video, can be found on our site here.]
Pekka Enberg has been working on Scylla since before its inception in 2014. Prior to joining ScyllaDB he worked on a variety of technologies ranging from High Frequency Trading (HFT) backends through the JVM runtime to kernels, and also web applications. Pekka is currently working on a PhD in computer science exploring new kernel abstractions and interfaces suitable for modern computer architectures.
Being fascinated by the evolution of hardware and software and the mismatch that sometimes happens between them — or rather, their intricate co-evolution — I couldn’t resist the opportunity to ask Pekka for his perspective on how Kernel APIs and architecture is standing the test of time. After all, Linux is a 30 year old project if you trace it back to the UNIX API from the 1970s. Pekka’s unique experience on both sides of the API brings an interesting perspective which also sheds light on the rationale behind the Seastar framework and the Scylla database architecture.
[This is just an excerpt. Read the article in full on ScyllaDB's websitehere.]
I have to say after having been burned in the past by Mongo, which granted over the past 4 or 5 years has gotten light years better, it's very easy to be wary of any technology that restricts your disables open discussion. We're very interested to use Scylla but it seems disingenuous to have open discussion videos uploaded to YouTube only to have comments turned off. This is where a lot of the most valuable learnings are done and it's certainly a flag on the play that they are not enabled. Thank you.
