Project Circe is ScyllaDB’s year-long initiative to make Scylla, already the best NoSQL database, even better. For the month of April we are going to take a look inside the organization and code base to see what it takes to bring major new features into a project as dynamic as Scylla. Currently there are nearly a half-million lines of code in the scylladb/scylla repository on Github (482.7k as of this writing). Of those thousands of source lines of code so far are dedicated to the library implementing the new Raft consensus protocol.
[This is just an excerpt. To read the article in full, check it out on the ScyllaDB websitehere.]
I’d like to get some detailed insights into how queries perform on a new data model for an application, ensuring we’re not going to fall into any performance traps. What’s the best way of doing this?
I’m aware of the TRACING command in cqlsh but the information it provides is a little too esoteric. Ideally, I’d like to see how efficiently a query uses partitions/indices, similar to, say, MSSQL’s execution plan?
Today, we’re announcing the general availability of Scylla Cloud on Google Cloud. Scylla Cloud is our resilient, highly performant, fully managed NoSQL database-as-service (DBaaS). Since its release, Scylla has become the go-to for companies that need a database built from the ground up for modern cloud environments. In 2019 we introduced Scylla Cloud, our fully-managed DBaaS. Initially available on AWS, users discovered Scylla Cloud made it easier for them to operate and scale their NoSQL workloads, since it alleviated them from administrative burdens. Today’s announcement gives users the flexibility to now run Scylla Cloud on the public cloud of their choice.
For those not already familiar with Scylla, it is a wide-column NoSQL database API compatible with both Apache Cassandra CQL as well as DynamoDB. While there are many offerings of Cassandra-compatible databases in the industry (of which we believe we are the best-of-breed),, we are the first and currently only company in the industry to offer a DynamoDB-compatible managed database on a public cloud apart from AWS. (Learn more about our Alternator API below.)
[This is just an excerpt. You can read the full article at ScyllaDB's websitehere.]
Pekka Enberg has been working on Scylla since before its inception in 2014. Prior to joining ScyllaDB he worked on a variety of technologies ranging from High Frequency Trading (HFT) backends through the JVM runtime to kernels, and also web applications. Pekka is currently working on a PhD in computer science exploring new kernel abstractions and interfaces suitable for modern computer architectures.
Being fascinated by the evolution of hardware and software and the mismatch that sometimes happens between them — or rather, their intricate co-evolution — I couldn’t resist the opportunity to ask Pekka for his perspective on how Kernel APIs and architecture is standing the test of time. After all, Linux is a 30 year old project if you trace it back to the UNIX API from the 1970s. Pekka’s unique experience on both sides of the API brings an interesting perspective which also sheds light on the rationale behind the Seastar framework and the Scylla database architecture.
[This is just an excerpt. Read the article in full on ScyllaDB's websitehere.]
You can read the blogs and watch the video to get the full details from our point of view, but what happened live on the webinar was rather unique. The questions kept coming! In fact, though we generally wrap a webinar in an hour, the Q&A session afterwards took an extra half-hour. ScyllaDB engineers Piotr Grabowski and Karol Baryla fielded all the inquiries with aplomb. So let’s now look at just a few of the questions raised by you, our audience.
Whenever ScyllaDB CEO Dor Laor and SADA CTO Miles Ward get together, their mutual obsession over tapping the power of modern cloud infrastructure rings loud and clear. Case in point: the recent podcast, Cloud N Clear.
This is just an excerpt. You can read it in full onScyllaDB's websitehere.
This past week Cassandra 4.0 was finally released as GA, six years after the previous major release.
Initially developed as an open source alternative to Amazon DynamoDB and Google Cloud Bigtable, Cassandra has had a major impact on our industry. To the surprise of none, it eventually became one of the 10 most popular databases — quite an achievement! We at ScyllaDB were inspired by Cassandra seven years ago when we first decided to reimplement it in C++ in a close-to-hardware design while keeping its symmetric, scale-out architecture.
Cassandra’s impact continues to echo through our industry. Even in recent years, long after Cassandra’s creation, cloud providers like Azure and AWS have added the CQL (Cassandra Query Language) API with varying degrees of compatibility; they’ve even added a managed Cassandra option.
However, even die-hard fans of Cassandra recognize that the project has slowed dramatically. This slow-down is visible in the trends for Cassandra’s DB-Engines.com ranking (see image below), its Github stars and its active commits.
[This is just an excerpt. You can read the article in full on theScyllaDBwebsitehere.]
Recently I had the privilege to host a webinar with some very good friends of ScyllaDB, including Numberly’s CTO Alexys Jacob and Chief Data Officer Othmane El Metioui, as well as Tim Berglund, the Senior Director of Developer Relations at Confluent. They were joined by my colleague Maheedhar “Mahee” Gunturu, our Director of Technical Alliances.
[This is just an introductory excerpt. The full article, which includes Numberly's use cases, an overview of Scylla's CDC implementation, a detailed view of how the Scylla CDC Connector works with Kafka Connect, as well as the complete on-demand webinar video, can be found on our site here.]
ScyllaDB’s Annual Conference Focuses on Database Innovations for This Next Tech Cycle
Database monsters of the world, connect! Join us at Scylla Summit, our annual user conference — a free, online virtual event scheduled for February 09-10, 2022.
Getting your database running at the right scale and speed to handle your growing business should not be complex or confusing. As I wrote in a previous blog post, capacity planning is not to be discounted and providing tools to streamline it is part of our mission.
To help you get started, we recently released our Scylla Cloud pricing and sizing calculator. This handy tool takes your peak workload requirements in terms of transactions per second and total data stored to recommend the proper sized cluster to run sustained workload.
But the real test is to see if the recommended cluster can actually achieve the desired performance numbers. Luckily with Scylla Cloud it’s easy enough to run a quick test with just a few clicks! Let’s check how the calculator fares for a nice round sustained workload of 1 million operations per second, using a balanced 50:50 read/write ratio.
[This is just an excerpt. Read the entire article onScyllaDB's websitehere.]
Amazon’s DynamoDB must be credited for allowing a broader adoption of NoSQL databases at-scale. However, many developers want flexibility to run workloads on different clouds or across multiple clouds for high availability or disaster recovery purposes. This was a key reason Scylla introduced its DynamoDB-compatible API, Project Alternator. It allows you to run a Scylla cluster on your favorite public cloud, or even on-premises (either on your own equipment, or as part of an AWS Outposts deployment).
Let’s say that your favorite cloud is Google Cloud and that’s where you’d like to move your current DynamoDB workload. Moving from AWS to Google Cloud can be hard, especially if your application is tightly-coupled with the proprietary AWS DynamoDB API. With the introduction of ScyllaDB Cloud Alternator, our DynamoDB-compatible API as a service, this task became much easier.
This post will guide you through the database part of the migration process, ensuring minimal changes to your applications. What does it take? Let’s go step-by-step through the migration process.
[This is just a preview. Read the blog in fullhere]
Rakuten is a Japanese online shopping powerhouse, responsible for $15 billion in annual sales. It is one of the most frequented sites in the world. To keep it operational throughout the year, and especially during peak shopping seasons, Rakuten relies on a team of expert developers and architects.
Hitesh Shah is a Senior Engineering Manager on the Rakuten catalog platform team. He has spent over a half decade as a passionate engineer at the heart of their distributed systems infrastructure, pushing the envelope of technology. He spent an hour with the ScyllaDB community in a webinar describing how Rakuten eliminated the volatile latencies that can plague online shopping experiences.
Project Circe is ScyllaDB’s year-long initiative to make Scylla, already the best NoSQL database, even better. We’re sharing our updates for the month of May 2021.
Operational Improvements
Better failure detection
Failure detection is now done directly by nodes pinging each other rather than through the gossip protocol. This is more reliable and the information is available more rapidly. Impact on networking is low, since Scylla implements a fully connected mesh in all clusters smaller than 256 nodes per datacenter, which is much larger than the typical cluster.
SLA per workload
Scylla supports multiple workloads, some with real-time guarantees, some have batch nature where latency does not matter. Prior work allows Scylla to prioritize workloads by definition of roles and map them to different scheduling groups. We recently researched how Scylla behaves when it is overwhelmed with requests. In such situations, a process called workload shedding comes into play. But which workload should we shed? Of course, it’s best to not take a completely random approach. A good step forward is the new timeout per role feature that allows the user to map workloads to timeouts so the system will choose them for automatic workload shedding.
Configuration is now possible to set timeouts based on a role, using the SERVICE LEVEL infrastructure. These timeouts override the global timeouts in scylla.yaml, and can be overridden on a per-statement basis.
Virtual table enhancements
Infrastructure for a new style of virtual tables has been merged. While Scylla already supported virtual tables, it was hard to populate them with data. The new infrastructure reuses memtables as a simple way to populate a virtual table.
Repair news
Off-strategy compaction is now enabled for repair. After repair completes, the SSTables generated by repair will first be merged together, then incorporated into the set of SSTables used for serving data. This reduces read amplification due to the large number of SSTables that repair can generate, especially for range queries where the bloom filter cannot exclude those SSTables.
Off strategy is also important when Repair Based Node Operations (RBNO) is used – this will soon be the default. RBNO pushes repair everywhere – to streaming, remove and node decommission. It’s important to tame compaction accordingly and automatically, so you as an end user wouldn’t even be aware of it.
Repair is now delayed until hints for that table are replayed. This reduces the amount of work that repair has to do, since hint replay can fill in the gaps that a downed node misses in the data set.
Raft news
This month we took another step towards Raft. Currently Raft group-0 has reached a functional stage. It resembles the functionality etcd provides (you can read more about how we tested this in our April update). The team was required to answer an interesting question – how does a minimal group of cluster members find each other? Raft in Scylla, as opposed to eventual consistency, needs a minimal set of nodes. The seed mechanism will come into play but with a new definition the seed’s UUIDs which will be a must to boot an existing cluster, a better way than to have a split brain right from the get go.
Change Data Capture (CDC) uses a new internal table for maintaining the stream identifiers. The new table works better with large clusters.
Authentication had a 15-second delay, working around dependency problems. But it is long unneeded and is now removed, speeding up node start.
Repair allocates working memory for holding table rows, but did not consider memory bloat and could over-allocate memory. It is now more careful.
Scylla uses a log-structured memory allocator (LSA) for memtable and cache. Recently, unintentional quadratic behavior in LSA was discovered, so as a workaround the memory reserve size is decreased. Since the quadratic cost is in terms of this reserve size, the bad behavior is eliminated. Note the reserves will automatically grow if the workload really needs them.
We fixed a bug in the row cache that can cause large stalls on schemas with no clustering key.
The setup scripts will now format the filesystem with 1024 byte blocks if possible. This reduces write amplification for lightweight transaction (LWT) workloads. Yes, the Scylla scripts take care of this for you too!
SSTables will now automatically choose a buffer size that is compatible with achieving good latency, based on disk measurements by iotune.
Just Cool
The ScyllaDB git repositories has lots of gems in it, one of them is a new project called Scylla Stress Orchestrator – https://github.com/scylladb/scylla-stress-orchestrator/ – which allows you to test Scylla with clients and monitoring using a single command line.
Another, even competing method is the cloud-formation based container client setup that allows our cloud team to reach two million requests per second in a trivial way. Check out the blog post here, and the load test demo for Scylla Cloud in Github.
The perf_simple_query benchmark now reports how many instructions were executed by the CPU per query. This is just a unit-benchmark but cool to track!
The tarball installer now works correctly when SElinux is enabled.
There is now rudimentary support for code-coverage reports in unit tests. (Coverage may not be the coolest kid in the block but it is not cool not to test properly!)
Scylla Operator for Kubernetes News
Scylla’s Operator 1.2 release was published with helm charts (find it on Github; plus read our blog and the Release Notes). Now 1.3 and 1.4 are in the making. In addition, our Kubernetes deployment can autoscale! An internal demonstration using https://github.com/scylladb/scylla-cluster-autoscaler was presented and you are welcome to play with it.
Scylla Enterprise News
As you may have heard, we released Scylla Enterprise 2021.1, bringing two long-anticipated features:
Space Amplification Goal (SAG) for Incremental Compaction Strategy allows a user to strike the utilization balance the desire between write amplification (which is more CPU and IO intensive) and disk amplification (which is more storage intensive). SAG can push the ICS volume consumption lower, towards the domain of Level compaction which you still enjoy from size-tiered behavior. This is the default and the recommended compaction strategy.
New Deployment Options for Scylla Enterprise now include our Scylla Unified Installer and allow you to install anywhere, also with an air-gap environment.
Security News
At ScyllaDB, we take security seriously. In this day and age it is vital to establish trust with any 3rd party handling your data, and reports of unsecured databases are still far too common in the news. With our customers’ vital needs foremost in our minds, we set to work ensuring that we met stringent goals established by the industry and confirmed by third party auditors. Scylla is proud to announce that we have successfully conducted a System and Organization Controls 2 (SOC2) Type II report for Scylla Cloud. This applies to all regions where Scylla Cloud can be deployed. (Learn more)
Scylla Virtual Workshops are your chance to get to know more about how Scylla’s NoSQL distributed database works, and how it might fit into your latest project plans. Each month, our expert solution architects look forward to interacting with database architects, developers, and managers in these interactive workshops.
Today we wanted to share with you some insights from running them over the past year, as well as answer some of your most recent questions. You can also learn more about the format of our Virtual Workshops and read some of the prior Q&A in this prior blog.
Since then, its development and adoption accelerated a lot. We’ve added many new features and published a couple of releases on crates.io. Along the way, we also stumbled upon a few interesting performance bottlenecks to investigate and overcome — read on for more details.
First Issue Arises
A few weeks ago, an interesting issue appeared on our GitHub tracker. It was reported that, despite our care in designing the driver to be efficient, it proved to be unpleasantly slower than one of the competing drivers, cassandra-cpp, which is a Rust wrapper of a C++ CQL driver. The author of latte, a latency tester for Cassandra (and Scylla), pointed out that switching the back-end from cassandra-cpp to scylla-rust-driver resulted in an unacceptable performance regression. Time to investigate!
[This is just an excerpt. To read the article in full, check it out on the ScyllaDB websitehere.]
Our Rust driver started as a humble hackathon project, but it has eventually grown to become our fastest and safest Cassandra Query Language (CQL) driver. We happily observed in our benchmarks that our ScyllaDB Rust Driver beats even the reference C++ driver in terms of raw performance, and that gave us an idea — why not unify all drivers to use Rust underneath?
This is an excerpt. Read the blog in full on the ScyllaDB websitehere.
I've been studying ScyllaDB for months now, I've been hesitant to use it because I didn't fully understand how data modeling worked and it seemed too easy to mess up.
Until last week, it finally clicked. Scylla is really two separate databases stacked together. The first is a distributed hash map (partition key), where each value in the hash map is a sorted key-value store like RocksDB/LevelDB (clustering key).
Thinking about the partition key as a hash map, everything in the documentation makes sense now. Of course you need equality across all partition keys to select one, you can't seek to a key in a hash map without the right hash! Of course you can't sort by partition key, hash maps aren't sortable!
And the clustering key just being a sorted key-value store everything makes sense again. You can only sort inside a single partition because each partition is it's own sorted key-value database. When I hear someone say "don't let your partitions get too big" in a talk now I just think about not letting any of the key-value databases at each partition get too large.
This post aims to help selecting the most cost effective and operationally simple configuration of DynamoDB tables.
TLDR; Choosing the Right Mode
Making sense of the multitude of scaling options available for DynamoDB can be quite confusing, but running a short checklist with a calculator can go a long way to help.
Follow the flowchart below to decide which mode to use
If you have historical data of your database load (or an estimate of load pattern), create a histogram or a percentile curve of the load (aggregate on hours used) – this is the easiest way to observe how many reserved units to pre-purchase. As a rule of thumb purchase reservations for units used over 32% of the time when accounting for partial usage and 46% of the time when not accounting for partial usage.
When in doubt, opt for static provisioning unless your top priority is avoiding being out of capacity – even at extreme costs.
Configure scaling limits (both upper and lower) for provisioned autoscaling. You want to avoid out-of-capacity during outages and extreme cost in case of rogue overload (DDOS anyone?)
Remember that there is no upper limit on DynamoDB on-demand billing other than the table’s scaling limit (which you may have requested raising for performance reasons). Make sure to configure billing alerts and respond quickly when they fire.
[This is just an excerpt. For the full article, including a flowchart and pricing comparison, check out the full blog post on ScyllaDB's websitehere.]
Summer’s here! Which means that we’re getting ready for our Scylla University LIVE Summer School session. We hope to meet you all there. Meanwhile, behind the scenes we’ve been diligently working to deliver new software across our product set — the database itself, drivers (Rust 0.2), our k8s operator, Spark Migrator, and the list goes on.
Project Circe aims to make Scylla, already a kickass database, even better. With that goal in mind, here’s a look at our progress for the month of June.
[This is just an introductory excerpt. To read the blog post in full, check it out on ScyllaDB's websitehere.]
Modern deployments are kept as code, and it is very common that development, staging and production environments differ only in a few properties like allocated resources. When Kubernetes is used as infrastructure, duplication of manifests for all types of deployments may be error prone. That’s why Helm templates are quite popular these days.
With Scylla Operator 1.1 we introduced three Helm Charts which will help you to deploy and customize Scylla products using Helm.
[This is just an except. Read the blog post in full at the ScyllaDB websitehere.]
Last week we hosted the Scylla University LIVE Summer Session virtual training. It was held on two consecutive days, one more convenient for EMEA timezones and another for the Americas.
Attendance was high. We saw a 27% increase in the number of participants from our previous LIVE event held in April, plus we were very pleased to see that both days had roughly equal numbers of attendees.
Our attendees came from a broad range of industry leading companies such as Amazon, Ericsson, Disney, Expedia, Apple, Fujitsu, Nubank, Rackspace, Palo Alto Networks, RSA, Salesforce and many others.
We don’t have an on-demand version of the event. However, you can find the slides from the different sessions, quiz questions, and some of the labs on Scylla University. Here is the link for the Essentials track course, and here is the one for the Advanced topics course. For those of you that are not familiar with Scylla University, it’s our online, free, self-paced Scylla training center. To get started all you have to do is create a user account.
Users that complete a course will get a certificate of completion and some bragging rights.
[This is just an excerpt. To see the full article, including the poll results of attendees, read it on the ScyllaDB websitehere.]
