At first I thought it was a Groovy limitation that was causing my scripts to run out of heap space when I ran them against large tables. It turned out to be the default behavior of the MySQL JDBC driver though.

By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate, and due to the design of the MySQL network protocol is easier to implement. If you are working with ResultSets that have a large number of rows or large values, and can not allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.

The solution is to override three default settings. The first two override the default settings of the ResultSets that will be returned by the statement.
sql.setResultSetType( java.sql.ResultSet.TYPE_FORWARD_ONLY )
sql.setResultSetConcurrency( java.sql.ResultSet.CONCUR_READ_ONLY )

This one adds a closure that gets called every time a statement gets created.
sql.withStatement{ stmt -> stmt.setFetchSize( Integer.MIN_VALUE ) }

If a multiple-column index exists on col1 and col2, the appropriate rows can be fetched directly. If separate single-column indexes exist on col1 and col2, the optimizer tries to find the most restrictive index by deciding which index finds fewer rows and using that index to fetch the rows.

To demonstrate this I went and got some historical stock price data from this site. After that, I created a test table and added some indexes in order to test some concepts.

CREATE TABLE stockdata (
   sdate DATE NOT NULL ,
   ticker VARCHAR( 4 ) NOT NULL ,
   open DECIMAL( 8,2 ) NOT NULL ,
   high DECIMAL( 8,2 ) NOT NULL ,
   low DECIMAL( 8,2 ) NOT NULL ,
   close DECIMAL( 8,2 ) NOT NULL ,
   volume INT NOT NULL
) ENGINE = MYISAM ;

load data infile 'sp500hst.txt'
into table stockdata
fields terminated by ',';

create index ix_sdate on stockdata ( sdate );
create index ix_ticker on stockdata ( ticker );

With the data loaded and indexes created on the ticker field and the sdate field, we’re ready to take a look at why this is important. Here’s the query that we’re going to use to test with:

explain extended
select *
from stockdata
where ticker = 'DIS'
and sdate between
   '2007-02-01' and '2007-02-28';

+----+-------------+-----------+------+--------------------+-----------+---------+-------+------+-------------+
| id | select_type | table     | type | possible_keys      | key       | key_len | ref   | rows | Extra       |
+----+-------------+-----------+------+--------------------+-----------+---------+-------+------+-------------+
|  1 | SIMPLE      | stockdata | ref  | ix_sdate,ix_ticker | ix_ticker | 6       | const |  245 | Using where |
+----+-------------+-----------+------+--------------------+-----------+---------+-------+------+-------------+

According to the results of our EXPLAIN, the optimizer chose to use the index on the ticker field, but not the index on the sdate field. It chose that index because there are only 253 entries in the data with a ticker of ‘DIS’, but there are 9501 entries with an sdate in Feb 2007. Basically, the plan is for it to grab all 253 ‘DIS’ entries, and then it will apply a filter to them in order to find the ones that are within the date range that we are looking for. Note that under ‘rows’, the optimizer thinks that this index will return 245 rows instead of 253.

This isn’t exactly ideal. It would be nice to have the database utilize indexes on both those fields. Since the database can only use one index per join, we will have to rely on a multiple-column ( aka compound ) index to get the desired performance.

drop index ix_sdate on stockdata;
drop index ix_ticker on stockdata;

create index ix_compound on stockdata( sdate, ticker );

+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
| id | select_type | table     | type  | possible_keys | key         | key_len | ref  | rows | Extra       |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
|  1 | SIMPLE      | stockdata | range | ix_compound   | ix_compound | 9       | NULL | 8872 | Using where |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+

According to the optimizer it’s going to have to filter through 8872 rows. This appears to be less efficient than just having the index on the ticker field. Why? Because the order of the columns in a multiple-column index is very important and should be from most selective to least. In this example, we’ve ordered them just the opposite. Instead, let’s drop that index and recreate it with the fields in the opposite order:

drop index ix_compound on stockdata;
create index ix_compound on stockdata( ticker, sdate );

+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
| id | select_type | table     | type  | possible_keys | key         | key_len | ref  | rows | Extra       |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
|  1 | SIMPLE      | stockdata | range | ix_compound   | ix_compound | 9       | NULL |   14 | Using where |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+

Down to an estimated 14 rows. The query actually returns 19. By utilizing a multiple-column index and carefully selecting the order of the columns in the index, I think this is about the best performance that we can get out of this query by just adjusting the indexes. I’m certain that there are other optimizations that could be performed. For instance, I could have done a better job structuring the table. Different data types would probably occupy less space on disk, require fewer IO reads, etc … That said, playing with these indexes probably gets us the biggest bang for the buck.

CREATE TABLE `order` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`name` VARCHAR( 20 ) NOT NULL
) ENGINE = INNODB ;

CREATE TABLE `suborder` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`order_id` INT NOT NULL,
`name` VARCHAR( 20 ) NOT NULL
) ENGINE = INNODB ;

CREATE TABLE `infraorder` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`suborder_id` INT NOT NULL,
`name` VARCHAR( 20 ) NOT NULL
) ENGINE = INNODB ;

insert into `order` ( name ) values ( 'Primates' );

insert into `suborder` ( order_id, name ) values ( 1, 'Strepsirrhini' );
insert into `suborder` ( order_id, name ) values ( 1, 'Haplorrhini' );

insert into `infraorder` ( suborder_id, name ) values ( 1, 'Lemuriformes' );
insert into `infraorder` ( suborder_id, name ) values ( 1, 'Chiromyiformes' );
insert into `infraorder` ( suborder_id, name ) values ( 1, 'Lorisiformes' );
insert into `infraorder` ( suborder_id, name ) values ( 2, 'Tarsiiformes' );
insert into `infraorder` ( suborder_id, name ) values ( 2, 'Simiiformes' );

Query analysis is done using the EXPLAIN statement. Here’s an example of how to use it:

explain extended
select o.name, s.name, i.name
from `order` o, suborder s, infraorder i
where o.id = s.order_id
and s.id = i.suborder_id
and i.name = 'Simiiformes';

The output looks like this:

+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
|  1 | SIMPLE      | o     | ALL  | PRIMARY       | NULL | NULL    | NULL |    1 |             |
|  1 | SIMPLE      | s     | ALL  | PRIMARY       | NULL | NULL    | NULL |    2 | Using where |
|  1 | SIMPLE      | i     | ALL  | NULL          | NULL | NULL    | NULL |    5 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
3 rows in set, 1 warning (0.01 sec)

It lists the tables in the order that they are joined in. The letters in the table column refer to the aliases that I declared in the query. It reads a row from the first table, then recursively finds a matching row in the second table, the third table, etc … In my example here it reads the one row from the order table first, then joins to the suborder table, and finally the infraorder table.

The ‘type’ column is probably the most important piece of data that is returned. The possible values, from best to worst, are system, const, eq_ref, ref, range, index, all. As you can see, our test query has returned nothing but ‘ALL’, the worst possible type of join in MySQL. ALL is basically a sequential scan, the optimizer is planning on reading every row in each table.

Let’s add a few indexes and see if we can’t improve this a little bit:

create index ix_subordid on infraorder ( suborder_id );
create index ix_ordid on suborder ( order_id );

+----+-------------+-------+------+------------------+----------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys    | key      | key_len | ref         | rows | Extra       |
+----+-------------+-------+------+------------------+----------+---------+-------------+------+-------------+
|  1 | SIMPLE      | o     | ALL  | PRIMARY          | NULL     | NULL    | NULL        |    1 |             |
|  1 | SIMPLE      | s     | ref  | PRIMARY,ix_ordid | ix_ordid | 4       | monkey.o.id |    1 |             |
|  1 | SIMPLE      | i     | ALL  | ix_subordid      | NULL     | NULL    | NULL        |    4 | Using where |
+----+-------------+-------+------+------------------+----------+---------+-------------+------+-------------+
3 rows in set, 1 warning (0.00 sec)

The join on the suborder table is now type ‘ref’. This is a good thing, it means that the query optimizer has chosen to use our index on the suborder table for every single row that gets returned from the query.

However, our index on the infraorder table isn’t being used. Why? I’m not entirely certain. I assume that it has to do with how little data there is in the table. If that’s the case, then the query optimizer assumes that returning all rows in order to satisfy the “i.name = ‘Simiiformes’” part of the where clause is more efficient than going to ix_subordid, and then only filtering on the names of those rows returned.

Let’s drop that index and create a multiple-column index instead:

drop index ix_subordid on infraorder;
create index ix_infra on infraorder ( name, suborder_id );

+----+-------------+-------+------+------------------+----------+---------+-------------------+------+--------------------------+
| id | select_type | table | type | possible_keys    | key      | key_len | ref               | rows | Extra                    |
+----+-------------+-------+------+------------------+----------+---------+-------------------+------+--------------------------+
|  1 | SIMPLE      | o     | ALL  | PRIMARY          | NULL     | NULL    | NULL              |    1 |                          |
|  1 | SIMPLE      | s     | ref  | PRIMARY,ix_ordid | ix_ordid | 4       | monkey.o.id       |    1 |                          |
|  1 | SIMPLE      | i     | ref  | ix_infra         | ix_infra | 26      | const,monkey.s.id |    2 | Using where; Using index |
+----+-------------+-------+------+------------------+----------+---------+-------------------+------+--------------------------+
3 rows in set, 1 warning (0.00 sec)

Now we’re using both the indexes in both of those tables. There’s an important caveat to using a multiple-column index though. In the example that I’ve created here, if I were to construct and execute a query on the infraorder table that referenced the name column but not the suborder_id, it would still use the index. The opposite is not true.

If you have a multiple-column index that has 3 columns, let’s call them A, B, and C. If your query uses all three, A and B, or just A, it can still utilize the index. If it uses just B, or just C, or B and C, it can’t. If it refers to A and C, then it will use the index for column A, but it will have to filter out the results that don’t match column C.

I hope that makes sense. I can’t think of any clearer way to explain the concept.

There are a few rules of thumb when it comes to indexes. These are almost universal, they don’t just apply to MySQL. First of all, you want an index that has short keys. This refers to the key_len column when we do an explain. The shorter the keys, the higher the number that will be read from the disk in each block. Another benefit is it means more of the index will be cached in memory.

Another rule of thumb is that you want your indexes to be as unique as possible. If you are indexing a column, you have a 100,000 rows in the table, and the column is comprised of four unique values, then your index isn’t much use.

MySQL supports indexing only part of the field. So, if you have a varchar(200), and the majority of the entries in the column are unique across the first 10 characters, you can index just those first 10 characters. Now, instead of using 200 characters for each key in the index, it only uses 10.

There is overhead associated with indexes. As a result, you should add them sparingly. This is especially true if you require any sort of performance when it comes to data modification queries (insert, update, delete). You don’t just have to modify the data, the indexes have to be modified as well.

What do you gain from using multiple-column indexes? They are more unique, and MySQL will favor the index that the query plan assumes will return the fewest number of rows. Another really cool thing, and most databases do this, if an index has all of the data necessary to return the answer without consulting the data, then the engine will use just the index. Examples are queries where you are trying to find a count of rows, a min or a max, etc … These are called covered queries and there are more possibilities if your index refers to more columns. Don’t go listing all the columns though. Then you just end up duplicating the table in the index space. Remember, small keys.

Two other things before I go to bed. I keep mentioning the query optimizer. Optimizer’s are a fascinating subject. They rely on statistical information about the database in order to make an educated guess about how to execute the query. The ‘ANALYZE TABLE‘ command analyzes the table and stores the key distribution information that the optimizer uses to make some of those decisions. It needs to be run periodically, in some cases frequently, in order for the optimizer to function properly.

Another important statement is ‘OPTIMIZE TABLE‘. If you use varchar, or any other variable length data type, then deleting data fragments your table. The OPTIMIZE command is a defragmenter for your database. It also sorts the index keys, which can improve performance.

Wow … sleepy now. If you are in the market for an effective sedative, then I recommend reading the MySQL 5.0 Reference Manual.