Hack MySQL

Query Response Time

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn MySQL query response time:

Performance is query response time.

This book explores that idea from various angles with a single intent: to help you achieve remarkable MySQL performance. Efficient MySQL performance means focusing on the best practices and techniques that directly affect MySQL performance—no superfluous details or deep internals required by DBAs and experts. I presume that you’re a busy professional who is using MySQL, not managing it, and that you need the most results for the least effort. That’s not laziness, that’s efficiency. To that end, this book is direct and to the point. And by the end, you will be able to achieve remarkable MySQL performance.

Indexes and Indexing

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn MySQL indexes and indexing:

Many factors determine MySQL performance, but indexes are special because performance cannot be achieved without them. You can remove other factors—queries, schemas, data, and so on—and still achieve performance, but removing indexes limits performance to brute force: relying on the speed and capacity of hardware. If this book were titled Brute Force MySQL Performance, the contents would be as long as the title: "Buy better, faster hardware." You laugh but just a few days ago I met with a team of developers who had been improving performance in the cloud by purchasing faster hardware until stratospheric costs compelled them to ask, "How else can we improve performance?"

MySQL leverages hardware, optimizations, and indexes to achieve performance when accessing data. Hardware is an obvious leverage because MySQL runs on hardware: the faster the hardware, the better the performance. Less obvious and perhaps more surprising is that hardware provides the least leverage. I explain why in a moment. Optimizations refer to the numerous techniques, algorithms, and data structures that enable MySQL to utilize hardware efficiently. Optimizations bring the power of hardware into focus. And focus is the difference between a light bulb and a laser. Consequently, optimizations provide more leverage than hardware. If databases were small, hardware and optimizations would be sufficient. But increasing data size deleverages the benefits of hardware and optimizations. Without indexes, performance is severely limited.

Data and Access Patterns

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn about MySQL data storage and access patterns:

Even when you master indexes and indexing, you will encounter queries that are simple and properly indexed but still slow. That’s when you begin to optimize around the query, starting with the data that it accesses. To understand why, let’s think about rocks.

Imagine that your job is to move rocks, and you have three piles of different sized rocks. The first pile contains pebbles: very light, no larger than your thumbnail. The second pile contains cobbles: heavy but light enough to pick up, no larger than your head. The third pile contains boulders: too large and heavy to pick up; you need leverage or a machine to move them. Your job is to move one pile from the bottom of a hill to the top. Which pile do you choose?

Sharding

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn about sharding MySQL:

On a single instance of MySQL, performance depends on queries, data, access patterns, and hardware. When direct and indirect query optimization—assiduously applied—no longer deliver acceptable performance, you have reached the relative limit of single-instance MySQL performance for the application workload. To surpass that relative limit, you must divide the application workload across multiple instances of MySQL to achieve MySQL at scale.

Sharding a database is the common and widely used technique of scaling out (or, horizontal scaling): increasing performance by distributing the workload across multiple databases. (By contrast, scaling up, or vertical scaling, increases performance by increasing hardware capacity.) Sharding divides one database into many databases. Each database is a shard, and each shard is typically stored on a separate MySQL instance running on separate hardware. Shards are physically separate but logically the same (very large) database.

Server Metrics and InnoDB

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn about MySQL server metrics, especially InnoDB metrics:

MySQL metrics are closely related to MySQL performance—that’s obvious. After all, the purpose of metrics in any system is to measure and report how the system is operating. What’s not obvious is how they are related. It’s not unreasonable if you currently see MySQL metrics as a black box with metrics inside that, in some way, indicate something about MySQL.

That view is not unreasonable (or uncommon) because MySQL metrics are often discussed but never taught. Even in my career with MySQL, I have never read or heard an exposition of MySQL metrics—and I have worked with people who created them. The lack of pedagogy for MySQL metrics is due to a false presumption that metrics do not require understanding or interpretation because their meaning is self-evident. That presumption has a semblance of truth when considering a single metric in isolation, like Threads_running: the number of threads running—what more is there to know? But isolation is the fallacy: MySQL performance is revealed through a spectrum of MySQL metrics.

Replication

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn the basics of MySQL replication, especially the cause of replication lag:

Replication lag is the delay between the time when a write occurs on a source MySQL instance and the time when that write is applied on a replica MySQL instance. Replication lag is inherent to all database servers because replication across a network incurs network latency.

Replication lag is data loss. Seriously. Do not dismiss replication lag.

Transactions and Data Locks

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn about MySQL transactions and data locks, which are closely related:

MySQL has non-transactional storage engines, but InnoDB is the default and the presumptive norm. Therefore, practically speaking, every MySQL query executes in a transaction by default, even a single SELECT statement.

From our point of view as programmers, transactions appear conceptual: BEGIN, execute queries, and COMMIT. Then we trust MySQL (and InnoDB) to uphold the ACID properties: atomicity, consistency, isolation, and durability. When the application workload—queries, indexes, data, and access patterns—is well optimized, transactions are a nonissue with respect to performance. (Most database topics are a nonissue when the workload is well optimized.) But behind the scenes, transactions invoke a whole new world of considerations because upholding ACID properties while maintaining performance is not an easy feat. Fortunately, MySQL shines at executing transactions.

Common Challenges

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn about common challenges when using MySQL:

This chapter is a short but important laundry list of common MySQL challenges and how to mitigate them. These challenges don’t fit into other chapters because most are not directly related to performance. But don’t underestimate them: the first two challenges, for example, can ruin a database. More importantly, these challenges are not special cases that only happen when the stars align and The Fates conspire to ruin your day. These are common challenges. Take them seriously, and expect to face them.

Excerpt from Efficient MySQL Performance, Chapter 9
Copyright 2021 Daniel Nichter
No reproduction of this excerpt without permission

Key Points

Split-brain occurs when two or more MySQL instances in the same replication topology are written to
Split-brain is a detriment to data integrity—the data can no longer be trusted
Data drift occurs when a replica becomes out of sync with the source
Data drift is real but the origin of the drift is virtually impossible to pinpoint
Data drift can be detected with pt-table-checksum
ORMs can generate very poor queries and overall performance
Schemas always change, so an online schema change (OSC) tool is must-have
There are three popular OSC tools: pt-online-schema-change, gh-ost, and Spirit
MySQL has non-standard SQL statements, options, and clauses
Applications do not fail gracefully by default; it takes effort to fail gracefully
High performance MySQL is difficult

Pitfalls

Not taking into account the key points above

Hack MySQL Articles

Additional Resources

Resource	Type	About
InnoDB and Online DDL	MySQL manual	Since schema changes are routine for developers, it’s important to understand how MySQL (and InnoDB) handle them because there are many edge cases depending on the specific type of change.
Spirit	Open source tool	The fastest, most advanced online schema change (OSC) change tool available.
gh-ost	Open source tool	The predecessor of Spirit. Both are solid tools, but use Spirit if possible.
pt-table-checksum	Open source tool	Still the only tool that can detection data drift on replicas

pt-online-schema-change
Before gh-ost and Spirit, pt-online-schema-change (pt-osc) was the de facto standard tool. It's still widely used and referenced today, but as the author of pt-osc I strongly advise that you use Spirit instead. The only reason to use pt-osc is for rare (and risky) edge cases that Spirit (and usually gh-ost) do not support for good reason.

Cloud Performance

Thu, 15 May 2025 06:47:00 -0700

Context

This is why you need to learn about the reality of using MySQL in the cloud:

MySQL in the cloud is fundamentally the same MySQL that you know and love (or know and tolerate). In the cloud, best practices and techniques are not only true but eminently true because cloud providers charge for every byte and millisecond of work. Performance is money in the cloud.

I wish it were as simple as “optimize the workload and you’re done”, but MySQL in the cloud raises unique considerations. The goal is to know and mitigate these cloud considerations so that you can focus on MySQL, not the cloud. After all, the cloud is nothing special: behind the proverbial curtain, it’s physical servers in a data center running programs like MySQL.

mlrd: DynamoDB-Compatible API on MySQL

Fri, 12 Dec 2025 16:39:00 -0500

Introducing mlrd (“mallard”) to the world: a DynamoDB-compatible API on MySQL. Crazy, but it works really well and I’m confident it will help a lot of businesses save a lot of money. Here’s why.

Learn MySQL Performance

Thu, 15 May 2025 06:47:00 -0700

Every year new cohorts of programmers, DBAs, managers, and more learn MySQL. After MySQL and HeatWave Summit 2025 a few weeks ago, I realized what job needs to be done for these people and how I could help them make progress: revamping this website to focus on “the path”.

An Efficient Path to Learn MySQL Performance

Mon, 21 Apr 2025 00:00:00 +0000

Conference Presentations

Video	Slides
MySQL Performance for Application Developers @ MySQL and HeatWave Summit 2025	mysql-summit-2025.pdf
MySQL Performance for Developers @ Percona Live 2023^†	PL23.pdf
Efficient MySQL Performance @ Percona Live 2022	PL22.pdf
Efficient MySQL Performance @ FOSDEM 2022	-

† Video recording not available

What Is the Path?

The path is an ordered sequence of MySQL topics to learn in order to understand and improve MySQL performance. It’s shown to the left (or below on small screens): start by learning topic 1 (query response time), then proceed topic by topic until the end, topic 9 (cloud performance).

Career Advice

Mon, 20 Jan 2025 00:00:00 +0000

‘Career Advice’ lists advice to help ambitious software engineers become the best in their field.

Monitoring Multi-threaded Replication Lag With Performance Schema

Sat, 28 Dec 2024 20:07:00 -0500

Used to be that replication lag was as simple as Seconds_Behind_Master (renamed to Seconds_Behind_Source). But with multi-threaded replication (MTR) this is no longer the case. It’s time to relearn replication lag monitoring using Performance Schema tables.

Replica Preserve Commit Order and Measuring Lag

Thu, 21 Nov 2024 19:01:00 -0500

With multi-threaded replication (MTR), a replica can commit transactions in the same order as the source, or not. This is determined by sysvar replica_preserve_commit_order (RPCO). As of MySQL v8.0.27 (released October 2021) it’s ON by default, but it was OFF by default for several years prior. In either case, it’s relatively new compared to 20+ years of single-threaded replication for which commit order was not an issue or option. But with MTR, it’s important to understand the affects of RPCO, especially with respect to the focus of this three-part series: replication lag.

Efficient MySQL Performance In 10 Sentences

Sat, 09 Nov 2024 14:27:00 -0500

Don’t have time to read Efficient MySQL Performance? Here’s the book (10 chapters) in one-liners.

Performance is query response time.
Proper left-most indexing is required for performance.
The less data, the better.
Access patterns (part of the workload) help or hinder performance.
Sharding is how to scale writes when single-node performance is truly reached.
Server metrics reflect how the app workload causes MySQL to work.
Replication lag is data loss.
Locks are held until a transaction commits, so commit quickly.
There are many other challenges that you might need to address—sorry.
MySQL in the cloud is slower and more expensive, so performance is more important than ever.

Group Commit and Transaction Dependency Tracking

Sun, 20 Oct 2024 22:35:00 -0400

MySQL 8.0 and newer change and improve how we measure and monitor replication lag. Even though multi-threaded replication (MTR) has been on by default for the last three years (since v8.0.27 released October 2021), the industry has been steeped in single-threaded replication for nearly 30 years. As a result, replication lag with MTR is a complicated topic because it depends on version, configuration, and more. This three-part series provides a detailed understanding, starting from what was originally an unrelated feature: binary log group commit.

Lessons From 20 Years Hacking MySQL (Part 1)

Thu, 20 Jun 2024 10:01:00 -0400

I vividly remember 2004 when I decided to specialize in MySQL because the year before I was homeless and living in my car. It’s been a long road and an amazing journey ever since.

Lessons From 20 Years Hacking MySQL (Part 2)

Thu, 20 Jun 2024 10:00:00 -0400

A relational database is more than a data bank, it’s a profound philosophical expression. This is the second and final part of “Lessons From 20 Years Hacking MySQL”.

Why MySQL Replication Is Fast

Fri, 31 May 2024 11:31:00 -0400

Replication being slow—replication lag—is a common complaint, but MySQL replication is actually really fast. Let’s run a controlled experiment and peek inside the Performance Schema and binary logs to see why.

When COMMIT Is the Slowest Query

Wed, 17 Apr 2024 16:06:00 -0400

When COMMIT is the slowest query, it means your storage is slow. Let’s look at an example.

The Fallacy of Small Functions

Sun, 24 Mar 2024 17:00:00 -0400

Decades of computer programming advice, best practices, and books have taught that small functions are good. That’s misleading at best and counterproductive at worst: increasing complexity rather than reducing it.

For brevity, I’m going to say things are false or wrong without explaining because you already have a fantastic explanation: A Philosophy of Software Design by John Ousterhout. Everything I’m arguing here is explained in that book.

Here’s an example of bad design:

The Infamous ORDER BY LIMIT Query Optimizer Bug

Tue, 28 Nov 2023 17:39:00 -0500

Which is faster: LIMIT 1 or LIMIT 20? Presumably, fetching less rows is faster than fetching more rows. But for 16 years (since 2007) the MySQL query optimizer has had a “bug”† that not only makes LIMIT 1 slower than LIMIT 20 but can also make the former a table scan, which tends to cause problems. This happened last week where I work, and although MySQL DBAs are familiar with this bug, I’m writing this blog post for developers to more clearly illustrate and explain what’s going on and why because it’s really counterintuitive.

The Future of MySQL Schema Change: Spirit

Mon, 06 Nov 2023 09:00:00 -0500

Using pt-online-schema or gh-ost? Of course you are; everyone in the MySQL industry does. But now there’s a new online schema change tool that will obsolete these two: Spirit by renowned MySQL expert Morgan Tocker.

Deferred Join: A Deep Dive

Wed, 09 Aug 2023 13:27:00 -0400

Deferred join is powerful. Deferred join is simple. Deferred join is misunderstood.

Announcing Finch: A New MySQL Benchmarking Tool

Tue, 01 Aug 2023 13:41:00 -0400

I’m happy to announce Finch: a new MySQL benchmarking tool for experts, developers, and modern infrastructure. TL;DR: https://square.github.io/finch/

Benchmarking

Sun, 28 May 2023 00:00:00 +0000

A primer on the fundamentals of database benchmarking: results, workload, benchmark types, cheats, challenges, and reporting

Are Aurora Performance Claims True?

Fri, 12 May 2023 09:38:00 -0400

Amazon claims that Aurora has “Up to 5X the throughput of MySQL”. Is it true? It wasn’t easy to find the truth, but I kept digging until I found it. This is a long read; let’s chase the rabbit all the way down the hole.

Announcing Blip: A New MySQL Monitor

Tue, 28 Mar 2023 13:40:00 -0400

Blip is a new MySQL monitor that collects and reports server metrics. But wasn’t this problem solved long ago? Not really…

The First Design

Fri, 03 Feb 2023 16:30:00 -0500

When I review the code for a new program, I often provide the original authors a lot of feedback. And just as often, those authors are a little annoyed by me, the feedback, or both. But I keep doing it for two reasons.

Level Up

Thu, 19 Jan 2023 00:00:00 +0000

‘Level Up’ explains how software engineers can become the best in their field and produce more output than most.

COMMIT Latency: Aurora vs. RDS MySQL 8.0

Mon, 02 Jan 2023 20:18:00 -0500

Let’s examine COMMIT latency on Aurora v2 (MySQL 5.7) vs. Aurora v3 (MySQL 8.0) vs. RDS MySQL 8.0 2-AZ vs. RDS MySQL 8.0 3-AZ “cluster”.

Blip: A New Open Source MySQL Metrics Collector

Thu, 22 Dec 2022 12:55:00 -0500

Blip is a new open source MySQL metrics collector, or “MySQL monitor” for short. But isn’t collecting MySQL metrics easy? And don’t we already have some open source MySQL monitors? Let’s take a trip down memory lane…

How to Write Well

Sun, 18 Dec 2022 00:00:00 +0000

‘How to Write Well’ is a short guide for software engineers on how to write well.

MySQL IOPS for Reads and Surprsies

Sun, 20 Nov 2022 13:00:00 -0500

When you think about IOPS, you probably think about writes because MySQL write I/O has a long tradition of optimization, benchmarking, new algorithms, new storage engines, and so forth. There’s no shortage of material on MySQL write I/O; just two examples from Percona are Scaling IO-Bound Workloads for MySQL in the Cloud and Tuning MySQL/InnoDB Flushing for a Write-Intensive Workload. But in this short blog post I highlight two other, less common aspects of MySQL I/O: reads and surprises.

Is MySQL in the Cloud the End of the DBA?

Sat, 15 Oct 2022 16:31:00 -0400

No, I don’t think so. But it is does change our profession and have important implications for software engineers using (not managing) MySQL.

How Not to Use MySQL

Fri, 16 Sep 2022 16:01:00 -0400

Chapter 9 of Efficient MySQL Performance changed in development. Originally, it was a chapter titled “Not MySQL”, as in “how not to use MySQL.” But we (O’Reilly and I) pulled the chapter, and the current chapter 9 in print is “Other Challenges”: an important laundry list of other challenges engineers using MySQL must be aware of and address. This blog post is a sketch of the unwritten chapter 9: how not to use MySQL.

Mining the MySQL Performance Schema for Transactions

Sun, 28 Aug 2022 13:29:00 -0400

The MySQL Performance Schema is a gold mine of valuable data. Among the many nuggets you can extract from it is an historical report of transactions: how long a transaction took to execute, what queries were executed in it (with query metrics), and idle time between queries. Mining this information is not trivial, but it’s fun and this blog post shows how to start.

MySQL Transaction Reporting

Sun, 28 Aug 2022 13:28:00 -0400

How to inspect and report MySQL transactions using the Performance Schema

Better Replication Heartbeats

Thu, 28 Jul 2022 13:57:00 -0400

We’ve been measuring MySQL replication lag with heartbeats for more than a decade. It works, but can we do better? Let’s see.

InnoDB Page Flushing Diagram

Sat, 18 Jun 2022 19:09:00 -0400

Who dares diagram a system and process as complex as InnoDB page flushing? I do.

NewDB: Fate of an Open Source Database

Sat, 21 May 2022 15:10:00 -0400

Recently, I’ve had several discussions with engineers, industry experts, and business people of various titles and functions (from sales to CEO) about the fate of a new open source database. It’s not MySQL, and I won’t name the real database because it doesn’t matter for the purpose of this post. But to make writing and communication more clear, let’s call the database “NewDB” and the business “New Corp.”

This blog post is a think piece on how New Corp. can win the market with NewDB by remaining true to its open source origins and build a profitable business.

I Don't Want to Shard (MySQL)

Fri, 20 May 2022 19:00:00 -0400

Chapter 5 of Efficient MySQL Performance addresses sharding, and it was difficult to write but not for technical reasons. Let me say a little more on the matter.

Access Patterns for MySQL

Sun, 17 Apr 2022 13:50:00 -0400

Access patterns intrigue me because it seems that everyone knows what they are and talks about them, but there’s also very little written about them—in MySQL literature, at least. That’s why I set out to enumerate a list of access patterns (specific to MySQL). Since there’s no apparent standard for access patterns, I cannot say how my list measures up, but after spending most my career with MySQL, I know this: it is necessary to consider these access patterns when evaluating and improving MySQL performance. Simply put: you cannot ignore how the application accesses MySQL.

Go Single-character Names

Fri, 25 Mar 2022 16:55:00 -0400

‘Go Single-character Names’ details the proper, idoimatic usage of single character symbols in Go.

Performance Is Less

Sat, 19 Mar 2022 16:04:00 -0400

Is MySQL performance about “more” or “less”? From the title, you can tell that I’m going to argue that it’s about “less”. Here’s the punchline: zero is maximum performance. Let’s see where this philosophical blog post leads us.

MySQL EXPLAIN ANALYZE

Sun, 27 Feb 2022 14:28:00 -0500

As of MySQL 8.0.18, EXPLAIN ANALYZE is an indispensable tool for understanding query execution because it breaks down the query execution stage of response time by measuring each step of the query execution plan. The information is illuminating, but the output is not intuitive: it requires practice and some understanding of how MySQL executes queries beyond the table join order shown by traditional EXPLAIN output. This blog post closely examines three different examples of EXPLAIN ANALYZE output.

Configuring MySQL Query Metrics

Sun, 16 Jan 2022 15:40:00 -0500

Editors and technical reviewers suggested that I cover how to configure MySQL query metrics in chapter 1 of Efficient MySQL Performance, but I deferred because it was out of scope for the book, which focuses on engineers using MySQL, not DBAs. As such, there’s only a note in chapter 1 that says: “Ask your DBA or read the MySQL manual.” But I’ll cover the topic here because that’s what this blog post series is for: behind the book.

Efficient MySQL Performance

Mon, 27 Dec 2021 09:30:00 -0500

After 17 years with MySQL, I wrote a book: Efficient MySQL Performance.

I’ll make a bold claim: a MySQL book like this has never been written—not even close. The preface explains why this book is unique:

Database Operations Manual

Sat, 16 Oct 2021 00:00:00 +0000

‘Database Operations Manual’ outlines requirements for operating a database server in production.

MySQL LRU Flushing and I/O Capacity

Thu, 02 Sep 2021 18:11:00 -0400

InnoDB background LRU list flushing is not limited by innodb_io_capcity or innodb_io_capacity_max. I’ll prove it in this blog post, but since MySQL experts disagree (or don’t know for sure), I’d like you to prove me wrong. This is not an intro; you’ll need to know all the InnoDB details wrt page flushing.

MySQL Data Locks: Mapping 8.0 to 5.7

Mon, 26 Jul 2021 15:40:00 -0400

As of MySQL 8.0, performance_schema.data_locks shows InnoDB data locks. Before MySQL 8.0, you must SET GLOBAL innodb_status_output_locks=ON and ruminate on the output of SHOW ENGINE INNODB STATUS. The image below (click to see full size) shows how the former maps to the latter for three record locks and one table lock on table t.

Information Schema tables INNODB_LOCKS and INNODB_LOCK_WAITS are deprecated as of MySQL 5.7 and removed as of MySQL 8.0. They are better than nothing, but “Persistence and Consistency of InnoDB Transaction and Locking Information” cautions their usage. Moreover, INNODB_LOCKS only shows locks blocking other transactions, which makes it unsuitable to examine InnoDB row locking without a blocking transaction.

MySQL Password Rotation with AWS Secrets Manager and Lambda

Mon, 28 Dec 2020 09:56:00 -0500

How to set up MySQL password rotation with AWS Secrets Manager, AWS Lambda, and Terraform

Query Metrics Requirements for MySQL

Sun, 01 Nov 2020 20:25:00 -0500

Let’s answer a question which, to my knowledge, has never been systematically addressed: What are the requirements for a production-ready query metrics app at scale? I am uniquely qualified to answer that because I have written four query metrics apps, two of which are the open-source standard for MySQL: pt-query-digest and the original (v1) code behind the query metrics/analysis part of Percona Monitoring and Management. I’ve also published a couple of packages related to query metrics: go-mysql/slowlog and go-mysql/query.

What the Flush?

Sun, 05 Jul 2020 18:50:00 -0300

Yves Trudeau and Francisco Bordenave, MySQL experts at Percona, recently published a three-part post: (1) Give Love to Your SSDs – Reduce innodb_io_capacity_max!; (2) InnoDB Flushing in Action for Percona Server for MySQL; (3) Tuning MySQL/InnoDB Flushing for a Write-Intensive Workload. It’s a fantastic read from start to finish, and it made me realize: dirty pages sound bad, but they are good. I suspect the opposite (“dirty pages are not good”) is a misconception due to an incomplete picture. Let’s complete the picture.

21 Parameter Group Values to Change in Amazon RDS for MySQL

Mon, 29 Jun 2020 11:15:00 -0300

Amazon RDS for MySQL uses many default values for system variables, but it also sets a few “sys vars” with different values. As with any database, neither product (MySQL) nor provider (AWS) defaults can best suite all use cases. It’s our responsibility to carefully review and set every important system variable. This is tedious and difficult, but I’ve done it for you.

Below are are 21 MySQL 5.7 system variables that I recommend changing by creating a new parameter group. This presumes new RDS instances; some of these sys vars cannot be changed easily after provisioning MySQL. This only applies to RDS for MySQL, not Amazon Aurora.

Missing Writes with MySQL GTID

Sun, 31 May 2020 00:00:00 +0000

GTID-based replication makes managing replication topology easy: just CHANGE MASTER to any node and voilà. It doesn’t always work, but for the most part it does. That’s great, but it can hide a serious problem: missing writes. Even when MySQL GTID-based replication says, “OK, sure!”, which is most of the time, you should double check it.

MySQL Threads Running

Sat, 25 Apr 2020 16:18:00 -0300

Queries per second (QPS) measures database throughput, but it does not reflect how hard MySQL is working. The latter is measured by Threads_running, expressed as a gauge (whereas QPS is a rate). Before discussing Threads_running, let’s consider an analogy:

Queries Effect Performance

Sun, 12 Jan 2020 00:00:00 +0000

Queries effect database performance. That’s not a typo: “effect” not “affect”. The difference is not a word game but an important way to think about database performance.

Many things can affect performance. For example, if the system runs out of memory then starts swapping which causes high disk IO latency, that will negatively affect performance. But external factors like that notwithstanding (i.e. when hardware and MySQL are normal and stable), it’s important to understand that queries effect performance.

Fixing Ghosted GTIDs

Sun, 15 Dec 2019 00:00:00 +0000

MySQL auto-positioning is an integral part of replication with GTID, but it’s neither required nor guaranteed to work. It’s possible to enable GTIDs but disable auto-positioning, and it’s possible that one MySQL instance cannot auto-position on another even when GTIDs are used. The former (GTID on but auto-pos off) is an issue for another time. The latter is the topic of this post: when MySQL GTID auto-positioning fails—and how to fix it.

Response Time SLO

Sat, 29 Jun 2019 19:00:00 -0300

Response time percentiles are a ubiquitous measure of system performance. The P999 (99.9th percentile) is a high bar, so it’s a good metric to determine a service level objective (SLO). For example, my team runs a system with a P999 ≤ 800 millisecond SLO. The system meets the SLO, but here’s its P999 graph:

Looks bad, but the system is meeting the SLO. To understand why, let’s look at how percentiles are calculated. The Percentiles page on this site is a longer, more technical read about the subject, but for our purposes here the TL;DR is: a percentile is a high water mark. Consider a literal high water mark:

Path and Dimension Metrics

Fri, 12 Apr 2019 09:00:00 -0400

The two most common strategies for identifying and organizing data points that comprise metric time series are path and dimension. This post is geared towards engineers who have never been introduced to the subject. Implementing metrics from the ground up, or adding new metrics to an existing system, can be overwhelming in nontrivial systems. Understanding these two strategies simplifies one’s efforts.

Path-based Metrics

Path-based metrics are the historical norm:

host.prod1.disk-free.sda
host.prod1.disk-free.sdc
host.prod2.disk-free.sda
host.prod2.disk-free.sdc

Data points are identified by a path like the fully qualified path name of a file: /home/brian/success.log. Also like a file, the data point name has a separator character, usually . or /. So the examples above have four parts: host (fixed), a hostname (variable), disk-free (fixed), a drive (variable). Data points, like files, are uniquely identified by their full path. For example, file “success.log” is not unique because it can exist in any directory. Likewise, “disk-free.sda” is not unique because it can apply to any host.

Go Error Types

Mon, 25 Mar 2019 11:00:00 -0400

This is not a post about Go error handling, it’s a post about Go error “types”: three different ways to construct and use the built-in error interface type. error is deceptively simple. When used in the ways described here, it allows us to design meaningful errors and error handling in large Go programs. Errors should be first-class citizens because when things break (and they always do), errors are the first—and often only—information we have. So make them great.

Percentiles

Sun, 10 Feb 2019 00:00:00 +0000

‘Percentiles’ is a deep dive into how percentiles are calculated using nearest rank and linear interpolation.

Go Channel Red Flags

Sun, 16 Dec 2018 13:00:00 -0300

More Than One close()

When reviewing code that uses channels, first thing I do is search for close() on every channel. More than one close() is red flag because, unless all uses of the channel are carefully written and tested, it can lead to a panic on close of closed channel. It can also lead to strange behavior or subtle bugs because the code that didn’t close the channel isn’t aware the channel is closed, so send and receive on the closed channel don’t have the intended effect.

Component-based Software Design

Sun, 02 Dec 2018 00:00:00 +0000

Software systems are difficult to design because code does not delineate and organize itself. Even experienced engineers can struggle to design a system because there are many solutions but it’s not clear which is the best. Compounding the difficulty: engineers know that code can work fine even if, internally, it’s an unorganized highly-entangled mess. We want code to be well-organized, but we don’t know how to make it so. Component-based design is the solution.

When MySQL Goes Away

Sun, 22 Jul 2018 18:00:00 -0300

Handling MySQL errors in Go is not easy. There are a lot of MySQL server error codes, and the Go MySQL driver as its own errors, and Go database/sql has its own errors, and errors can bubble up from other packages, like net.OpError. Consequently, Go programs tend not to handle errors. Instead, they simply report errors:

err := db.Query(...).Scan(&v)
if err != nil {
 return err
}

And then the error is logged or reported somewhere. This is as poor as it common, and it’s extremely common. A robust program handles the error: retry the query if possible; or report a more specific error; else, report the unhandled error. But robust MySQL error handling in Go requires very specific knowledge and experience that is beyond the reasonable purview of app developers.

Thinking in Go

Sun, 13 May 2018 09:00:00 -0700

After learning Go, it is necessary to think in Go in order to design and implement well-written, idiomatic Go. This blog post outlines aspects that I think are important for thinking in Go. A full exploration of these aspects could fill a book, but this only an orientation for experienced programmers new to Go.

Programming languages have aspects which affect how we think, design, and implement software with the language: syntax, documentation, features/capabilities, language type (object-oriented, functional, compiled, etc.), etc. Consider Perl Time::HiRes module vs. Ruby Time class vs. Go Time package. Even subtle differences like “module” vs. “class” vs. “package” reflect thinking in the language. Effective language use requires correct language thinking. Applying Java thinking to Go results in non-idiomatic Go code, which is to say: Go code that does not “speak to” other Go programmers. This is not a superficial concern: from open-source to proprietary enterprise software, a codebase must be familiar and intuitive to other programmers with respect to the language so that the language itself does not impede the already difficult task of ongoing software maintenance by people who did not write the original code.

Bootloader Design Pattern

Sun, 04 Feb 2018 19:30:00 -0700

Extensibility is a lofty goal for software design, and difficult to achieve. The Wikipedia article notes:

Extensible design in software engineering is to accept that not everything can be designed in advance.

That’s a paradox because it entails that extensible design does not accept that extensibility can be designed in advance! Fortunately, there’s at least one solution that can be intentionally designed and implemented in advance. I call it the “Bootloader” design pattern. Its intention is twofold: boot the program and make it extensible to the caller. Of course, every program is booted (started) somehow, usually a main() function that does a lot and isn’t well tested. But before we dive into Bootloader, let’s consider Keurig.

Database Scalability: Contention and Crosstalk

Sun, 28 Jan 2018 11:20:00 -0700

New engineers might not know and experienced engineers might not believe that database systems do not and cannot scale gracefully to 100% system capacity but, rather and counterintutively, exhibit retrograde performance at some point less than 100% system capacity. The reasonable presumption is:

We presume that a database can scale to almost 100% of system capacity, and max performance is a small fraction at the very top (right, orange slice) because, after all, the system itself needs some resources to run itself. But the reality of database scalability is far less optimistic:

Crash-safe MySQL Replication

Tue, 23 Jan 2018 18:40:00 -0700

MySQL crash-safe replication is an old feature (~4 years as of MySQL 5.6), but it’s not consistently understood or applied. The MySQL manual on the topic, 16.3.2 Handling an Unexpected Halt of a Replication Slave, is correct and authoritative, but unless you grok MySQL replication that page doesn’t make it obvious why crash-safe replication works. Other blog posts explain why, but sometimes add other considerations, making it unclear which settings are necessary and sufficient. The aim of this blog post is total enlightenment, a full crash-safe-spiritual awakening. Light the censers and let us begin at the beginning…

Tech Workers Are Good People

Sun, 31 Dec 2017 09:50:00 -0700

The Other Tech Bubble is a provocative and interesting read. The author clearly has a lot of experience with Silicon Valley and has done their homework. There are many good insights, like “They’re still asking if it’s possible to do something, and not whether they should.” But there are two claims and one ambiguity that need to be addressed.

Villains

The first claim is:

In 2008, it was Wall Street bankers. In 2017, tech workers are the world’s villain.

Approaching a New Software Project

Sun, 10 Dec 2017 00:00:00 +0000

Approaching a new software project is difficult for two reasons. First, there’s no “guiding star”, no objective starting point. The developer can begin from and proceed in many directions, which makes choosing difficult. They want to proceed in a good direction from the start, not wasting time or effort, but how can they make a good choice without a guide or the benefit of experience? Being forced to choose and move ahead, the choice is often made randomly rather than methodically. If lucky, the choice is good, but as Ryan says: “Luck is not a strategy.” Second, this is not something commonly taught. It’s common for developers to learn by being thrown into water and they either sink or swim. Becoming good at moving from idea to design to implementation to the long tail of software development/maintenance is a skill that requires practice, learning from mistakes, and learning from other resources (books, blogs, people, etc.).

Ideas, Leaders, and Engineers

Sun, 12 Nov 2017 00:00:00 +0000

The Senior Engineer’s Guide to Helping Others Make Decisions is a good read and good advice. I would summarize the advice as:

“Seek first to understand, then to be understood” (Dr. Stephen Covey)
Lead and guide, don’t micromanage
Embrace change, which entails embracing other people’s thinking and way of doing things

In the first dialog, the senior engineer fails those three points.

Av’s blog post made me think…

Good Ideas

Let me jump right to the punch line: levels of seniority should not apply to ideas. Many years ago I realized the following, one of my personal maxims:

Five-Point Checklist for Excellent Software

Sun, 17 Sep 2017 00:00:00 +0000

I’ve never met a developer who hasn’t read or reviewed a lot of code written by other developers. If there’s one truism in the field of software of engineering it’s that the process is collaborative. Unless you write and never release software, other people will read the code, and they’re most likely reading it not for a fun bedtime story but because they need to understand and probably fix or modify it.

Designing Tools for Integration

Tue, 05 Sep 2017 00:00:00 +0000

When we write a new tool to do X, it’s common to program the tool to do X in one way. When X is trivial or very narrow in scope, this makes sense, and programming any more would fall prey to over-engineering. However, when the tool does many things (all logically related, else it falls prey to bloat and/or feature-creep), there quickly becomes many ways to accomplish those many things. No problem, we program all those things, too, but during my tool-making career I’ve learned two things:

How To Test the Database

Sun, 20 Aug 2017 11:20:00 -0700

“How do I test the database?” is a question I’ve been asked by colleagues many times. There’s a good, clean solution, but let’s first step back from what seems to be the problem to unpack why “testing the database” is not special (but nonetheless important). In other words: colleagues don’t ask, “How do I test such-and-such package?”, because that’s common and well understood. The question about testing the database implies that the database is a special beast that, if not treated properly, will devour the developer in their sleep.

Design Before Implementation

Tue, 01 Aug 2017 00:00:00 +0000

Like most software engineers, I review my colleagues’ code. I rarely provide feedback on implementation details because developers rarely choose obviously bad implementations. I focus my attention on design rather than implementation for one simple reason: implementation details are easy to change when software is well designed. Or, from the business perspective, design is the most costly aspect of software to change, so I review for great and therefore cost-effective design.

Reading os/exec.Cmd Output Without Race Conditions

Sat, 17 Jun 2017 00:00:00 +0000

Golang Weekly issue 164 features a nice article, Advanced command execution in Go with os/exec, which details several ways of working with os/exec, especially how to read STDOUT and STDERR while the command is running. This common task is commonly done wrong, where “wrong” means “the code has a race condition”. This blog post shows how to read STDOUT and STDERR from an os/exec.Cmd while it’s running and without race conditions.

Problem

The Internet is rife solutions like Krzysztof provides (but afaik no one has provided them in one place and so thoroughly, so his post is a good read nonetheless), but let’s test one:

Custom MongoDB Replica Set Write Concern

Sun, 11 Jun 2017 00:00:00 +0000

One of the great things about MongoDB is write concern:

Write concern describes the level of acknowledgement requested from MongoDB for write operations to a standalone mongod or to replica sets or to sharded clusters.

Awesome! I want 1 write ack from any secondary in the primary DC so when the primary crashes we’re guaranteed to have an up-to-date secondary in the primary DC ready to be elected primary. Here’s my production MongoDB replica set:

Go Antipatterns

Sat, 03 Jun 2017 18:51:14 -0700

‘Go Antipatterns’ describes bad Go code and how to make it better.

Idiomatic Go

Sat, 03 Jun 2017 00:00:00 +0000

‘Idiomatic Go’ is a list of resources for learning about idomatic Go.

MySQL Select and Sort Status Variables

Fri, 23 Dec 2005 00:00:00 -0500

A deep dive into the MySQL select and sort status variables

About

Mon, 01 Jan 0001 00:00:00 +0000

Hack MySQL is the personal website of Daniel Nichter, author of the book Efficient MySQL Performance published by O’Reilly. His webiste explores MySQL, Go, and software design.