At first I thought it was a Groovy limitation that was causing my scripts to run out of heap space when I ran them against large tables. It turned out to be the default behavior of the MySQL JDBC driver though.

By default, ResultSets are completely retrieved and stored in memory. In most cases this is the most efficient way to operate, and due to the design of the MySQL network protocol is easier to implement. If you are working with ResultSets that have a large number of rows or large values, and can not allocate heap space in your JVM for the memory required, you can tell the driver to stream the results back one row at a time.

The solution is to override three default settings. The first two override the default settings of the ResultSets that will be returned by the statement.
sql.setResultSetType( java.sql.ResultSet.TYPE_FORWARD_ONLY )
sql.setResultSetConcurrency( java.sql.ResultSet.CONCUR_READ_ONLY )

This one adds a closure that gets called every time a statement gets created.
sql.withStatement{ stmt -> stmt.setFetchSize( Integer.MIN_VALUE ) }

1. Persistence. If the network has an error, or I just happen to click the silly little x on my terminal window, then all I have to do is ssh back into the server and restart screen (screen -r). It will be like I never left. This means that I don’t have to worry about running everything with a nohup, and if I start a job at work I can go home and reconnect (screen -Dr) to my terminal session in order to continue monitoring the job.
2. Multiple windows. Ssh into the server, start up screen, start a job in that shell, type ‘Ctrl-a c’, and voila, a new shell in a new window. Now I can go and adjust the size and position of the windows. For instance, let’s say that I need to tail five different log files. I can start up five windows, resize and position them all on the screen, and then I’m monitoring all those files in one terminal session. I can switch back and forth between the windows by typing ‘Ctrl-a #’ where # is the number of the window, ‘Ctrl-a n’ for next, ‘Ctrl-a p’ for previous, or ‘Ctrl-a “‘ to get a list of all the windows that you can arrow up or down through. You can also rename the window you are in by typing ‘Ctrl-a A’.

Unfortunately, screen sucks straight out of the box. I recommend creating a .screenrc with these lines at a minimum:

startup_message off
shell bash

termcapinfo xterm "ks=E[?1lE:kuE[A:kd=E[B:kl=E[D:kr=E[C:kh=E[5~:kH=E[F"

caption always "%{= bb}%{+b w}%n %h %=%t %c"
hardstatus alwayslastline "%{-b gk}%-w%{+b kg}%50>%n %t%{-b gk}%+w%<"

" screen starts at window 0. These two lines make the first window 1, and
" binds 0 to window 10.
bind c screen 1
bind 0 select 10

After starting screen you should now have two lines at the bottom of your screen. The first should be blue and have the window number, name, and system time. The second should be green and have a listing of all the current windows and their names. I prefer a dark background and these colors work really well on top of black. You should review the screen manpage for more things that you can add to those status lines. One of my favorites is %l which displays the current load on the system.

Another thing that you can do with screen is create custom screenrc files for different situations. I have several that I use which automatically open certain windows for me. For instance, if I "cp .screenrc .mysqlscreenrc" and then add the following lines to the bottom of .mysqlscreenrc:

screen -t "database 1"   1 mysql -h hostname -u username -ppassword data1
screen -t "database 2"   2 mysql -h hostname -u username -ppassword data2
screen -t "localhost"    3 bash

Then, add this alias "alias mysqlscreen='screen -S mysql -c ~/.mysqlscreenrc'". Now all you have to do is type mysqlscreen when you log in, and it will automatically open three windows for you. Two of them will open up mysql monitor sessions, and the third will open up a shell. The -S part of the screen command names the screen session. That way if you become disconnected then a 'screen -ls' will be more meaningful, and it will be easier to know which session is which when you do your 'screen -r'.

4:32pm up 730 days, 22:25, 231 users, load average: 0.60, 0.63, 0.69

Update: Stupid Murphy and his stupid laws. Just spent an hour working on two other machines. Only solution appeared to be rebooting both machines.

There’s an old practical joke where people pry the keys off a keyboard and replace them out of order. A user who is not a touch typist can get really confused when they look for the right letters. The xmodmap command can be used for an excellent alternative to this prank. For example, you can add the following commands into the victim’s .bashrc file:xmodmap -e ‘keycode 91 = t T’
xmodmap -e ‘keycode 92 = s S’

This prank swaps the s and t keys on the keyboard without requiring any physical keyboard modifications. Alternately, if you can pry up and replace the keys on the keyboard, follow it up by remapping the keyboard so it actually works correctly!

Never thought of that before. Now I have no choice but to try it out.

A week ago a review of the Secustick was posted on Tweakers.net. They loaded the software on the stick into a debugger and learned that all someone had to do was place a breakpoint after a function called VerifyPassWord(), alter the return code from a 0 to a 1, and voilà! (Pun intended.) Anybody could get access to the contents of the drive.

It’s generally my belief that the free market is efficient. However, Bruce’s essay brings up a good point. Shoddy products like the Secustick have an advantage in the marketplace. Namely, they cost less to produce. Normally this isn’t an issue because the consumer has the skillset and the ability to evaluate that the clothing they are buying from Walmart isn’t as high a quality as what they would buy from JC Penney’s. How are you supposed to know when it comes to something like the Secustick? Or a firewall? Or a virus scanner? Or a <insert any suitably complicated technology>?

If a multiple-column index exists on col1 and col2, the appropriate rows can be fetched directly. If separate single-column indexes exist on col1 and col2, the optimizer tries to find the most restrictive index by deciding which index finds fewer rows and using that index to fetch the rows.

To demonstrate this I went and got some historical stock price data from this site. After that, I created a test table and added some indexes in order to test some concepts.

CREATE TABLE stockdata (
   sdate DATE NOT NULL ,
   ticker VARCHAR( 4 ) NOT NULL ,
   open DECIMAL( 8,2 ) NOT NULL ,
   high DECIMAL( 8,2 ) NOT NULL ,
   low DECIMAL( 8,2 ) NOT NULL ,
   close DECIMAL( 8,2 ) NOT NULL ,
   volume INT NOT NULL
) ENGINE = MYISAM ;

load data infile 'sp500hst.txt'
into table stockdata
fields terminated by ',';

create index ix_sdate on stockdata ( sdate );
create index ix_ticker on stockdata ( ticker );

With the data loaded and indexes created on the ticker field and the sdate field, we’re ready to take a look at why this is important. Here’s the query that we’re going to use to test with:

explain extended
select *
from stockdata
where ticker = 'DIS'
and sdate between
   '2007-02-01' and '2007-02-28';

+----+-------------+-----------+------+--------------------+-----------+---------+-------+------+-------------+
| id | select_type | table     | type | possible_keys      | key       | key_len | ref   | rows | Extra       |
+----+-------------+-----------+------+--------------------+-----------+---------+-------+------+-------------+
|  1 | SIMPLE      | stockdata | ref  | ix_sdate,ix_ticker | ix_ticker | 6       | const |  245 | Using where |
+----+-------------+-----------+------+--------------------+-----------+---------+-------+------+-------------+

According to the results of our EXPLAIN, the optimizer chose to use the index on the ticker field, but not the index on the sdate field. It chose that index because there are only 253 entries in the data with a ticker of ‘DIS’, but there are 9501 entries with an sdate in Feb 2007. Basically, the plan is for it to grab all 253 ‘DIS’ entries, and then it will apply a filter to them in order to find the ones that are within the date range that we are looking for. Note that under ‘rows’, the optimizer thinks that this index will return 245 rows instead of 253.

This isn’t exactly ideal. It would be nice to have the database utilize indexes on both those fields. Since the database can only use one index per join, we will have to rely on a multiple-column ( aka compound ) index to get the desired performance.

drop index ix_sdate on stockdata;
drop index ix_ticker on stockdata;

create index ix_compound on stockdata( sdate, ticker );

+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
| id | select_type | table     | type  | possible_keys | key         | key_len | ref  | rows | Extra       |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
|  1 | SIMPLE      | stockdata | range | ix_compound   | ix_compound | 9       | NULL | 8872 | Using where |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+

According to the optimizer it’s going to have to filter through 8872 rows. This appears to be less efficient than just having the index on the ticker field. Why? Because the order of the columns in a multiple-column index is very important and should be from most selective to least. In this example, we’ve ordered them just the opposite. Instead, let’s drop that index and recreate it with the fields in the opposite order:

drop index ix_compound on stockdata;
create index ix_compound on stockdata( ticker, sdate );

+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
| id | select_type | table     | type  | possible_keys | key         | key_len | ref  | rows | Extra       |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+
|  1 | SIMPLE      | stockdata | range | ix_compound   | ix_compound | 9       | NULL |   14 | Using where |
+----+-------------+-----------+-------+---------------+-------------+---------+------+------+-------------+

Down to an estimated 14 rows. The query actually returns 19. By utilizing a multiple-column index and carefully selecting the order of the columns in the index, I think this is about the best performance that we can get out of this query by just adjusting the indexes. I’m certain that there are other optimizations that could be performed. For instance, I could have done a better job structuring the table. Different data types would probably occupy less space on disk, require fewer IO reads, etc … That said, playing with these indexes probably gets us the biggest bang for the buck.

CREATE TABLE `order` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`name` VARCHAR( 20 ) NOT NULL
) ENGINE = INNODB ;

CREATE TABLE `suborder` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`order_id` INT NOT NULL,
`name` VARCHAR( 20 ) NOT NULL
) ENGINE = INNODB ;

CREATE TABLE `infraorder` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`suborder_id` INT NOT NULL,
`name` VARCHAR( 20 ) NOT NULL
) ENGINE = INNODB ;

insert into `order` ( name ) values ( 'Primates' );

insert into `suborder` ( order_id, name ) values ( 1, 'Strepsirrhini' );
insert into `suborder` ( order_id, name ) values ( 1, 'Haplorrhini' );

insert into `infraorder` ( suborder_id, name ) values ( 1, 'Lemuriformes' );
insert into `infraorder` ( suborder_id, name ) values ( 1, 'Chiromyiformes' );
insert into `infraorder` ( suborder_id, name ) values ( 1, 'Lorisiformes' );
insert into `infraorder` ( suborder_id, name ) values ( 2, 'Tarsiiformes' );
insert into `infraorder` ( suborder_id, name ) values ( 2, 'Simiiformes' );

Query analysis is done using the EXPLAIN statement. Here’s an example of how to use it:

explain extended
select o.name, s.name, i.name
from `order` o, suborder s, infraorder i
where o.id = s.order_id
and s.id = i.suborder_id
and i.name = 'Simiiformes';

The output looks like this:

+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key  | key_len | ref  | rows | Extra       |
+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
|  1 | SIMPLE      | o     | ALL  | PRIMARY       | NULL | NULL    | NULL |    1 |             |
|  1 | SIMPLE      | s     | ALL  | PRIMARY       | NULL | NULL    | NULL |    2 | Using where |
|  1 | SIMPLE      | i     | ALL  | NULL          | NULL | NULL    | NULL |    5 | Using where |
+----+-------------+-------+------+---------------+------+---------+------+------+-------------+
3 rows in set, 1 warning (0.01 sec)

It lists the tables in the order that they are joined in. The letters in the table column refer to the aliases that I declared in the query. It reads a row from the first table, then recursively finds a matching row in the second table, the third table, etc … In my example here it reads the one row from the order table first, then joins to the suborder table, and finally the infraorder table.

The ‘type’ column is probably the most important piece of data that is returned. The possible values, from best to worst, are system, const, eq_ref, ref, range, index, all. As you can see, our test query has returned nothing but ‘ALL’, the worst possible type of join in MySQL. ALL is basically a sequential scan, the optimizer is planning on reading every row in each table.

Let’s add a few indexes and see if we can’t improve this a little bit:

create index ix_subordid on infraorder ( suborder_id );
create index ix_ordid on suborder ( order_id );

+----+-------------+-------+------+------------------+----------+---------+-------------+------+-------------+
| id | select_type | table | type | possible_keys    | key      | key_len | ref         | rows | Extra       |
+----+-------------+-------+------+------------------+----------+---------+-------------+------+-------------+
|  1 | SIMPLE      | o     | ALL  | PRIMARY          | NULL     | NULL    | NULL        |    1 |             |
|  1 | SIMPLE      | s     | ref  | PRIMARY,ix_ordid | ix_ordid | 4       | monkey.o.id |    1 |             |
|  1 | SIMPLE      | i     | ALL  | ix_subordid      | NULL     | NULL    | NULL        |    4 | Using where |
+----+-------------+-------+------+------------------+----------+---------+-------------+------+-------------+
3 rows in set, 1 warning (0.00 sec)

The join on the suborder table is now type ‘ref’. This is a good thing, it means that the query optimizer has chosen to use our index on the suborder table for every single row that gets returned from the query.

However, our index on the infraorder table isn’t being used. Why? I’m not entirely certain. I assume that it has to do with how little data there is in the table. If that’s the case, then the query optimizer assumes that returning all rows in order to satisfy the “i.name = ‘Simiiformes’” part of the where clause is more efficient than going to ix_subordid, and then only filtering on the names of those rows returned.

Let’s drop that index and create a multiple-column index instead:

drop index ix_subordid on infraorder;
create index ix_infra on infraorder ( name, suborder_id );

+----+-------------+-------+------+------------------+----------+---------+-------------------+------+--------------------------+
| id | select_type | table | type | possible_keys    | key      | key_len | ref               | rows | Extra                    |
+----+-------------+-------+------+------------------+----------+---------+-------------------+------+--------------------------+
|  1 | SIMPLE      | o     | ALL  | PRIMARY          | NULL     | NULL    | NULL              |    1 |                          |
|  1 | SIMPLE      | s     | ref  | PRIMARY,ix_ordid | ix_ordid | 4       | monkey.o.id       |    1 |                          |
|  1 | SIMPLE      | i     | ref  | ix_infra         | ix_infra | 26      | const,monkey.s.id |    2 | Using where; Using index |
+----+-------------+-------+------+------------------+----------+---------+-------------------+------+--------------------------+
3 rows in set, 1 warning (0.00 sec)

Now we’re using both the indexes in both of those tables. There’s an important caveat to using a multiple-column index though. In the example that I’ve created here, if I were to construct and execute a query on the infraorder table that referenced the name column but not the suborder_id, it would still use the index. The opposite is not true.

If you have a multiple-column index that has 3 columns, let’s call them A, B, and C. If your query uses all three, A and B, or just A, it can still utilize the index. If it uses just B, or just C, or B and C, it can’t. If it refers to A and C, then it will use the index for column A, but it will have to filter out the results that don’t match column C.

I hope that makes sense. I can’t think of any clearer way to explain the concept.

There are a few rules of thumb when it comes to indexes. These are almost universal, they don’t just apply to MySQL. First of all, you want an index that has short keys. This refers to the key_len column when we do an explain. The shorter the keys, the higher the number that will be read from the disk in each block. Another benefit is it means more of the index will be cached in memory.

Another rule of thumb is that you want your indexes to be as unique as possible. If you are indexing a column, you have a 100,000 rows in the table, and the column is comprised of four unique values, then your index isn’t much use.

MySQL supports indexing only part of the field. So, if you have a varchar(200), and the majority of the entries in the column are unique across the first 10 characters, you can index just those first 10 characters. Now, instead of using 200 characters for each key in the index, it only uses 10.

There is overhead associated with indexes. As a result, you should add them sparingly. This is especially true if you require any sort of performance when it comes to data modification queries (insert, update, delete). You don’t just have to modify the data, the indexes have to be modified as well.

What do you gain from using multiple-column indexes? They are more unique, and MySQL will favor the index that the query plan assumes will return the fewest number of rows. Another really cool thing, and most databases do this, if an index has all of the data necessary to return the answer without consulting the data, then the engine will use just the index. Examples are queries where you are trying to find a count of rows, a min or a max, etc … These are called covered queries and there are more possibilities if your index refers to more columns. Don’t go listing all the columns though. Then you just end up duplicating the table in the index space. Remember, small keys.

Two other things before I go to bed. I keep mentioning the query optimizer. Optimizer’s are a fascinating subject. They rely on statistical information about the database in order to make an educated guess about how to execute the query. The ‘ANALYZE TABLE‘ command analyzes the table and stores the key distribution information that the optimizer uses to make some of those decisions. It needs to be run periodically, in some cases frequently, in order for the optimizer to function properly.

Another important statement is ‘OPTIMIZE TABLE‘. If you use varchar, or any other variable length data type, then deleting data fragments your table. The OPTIMIZE command is a defragmenter for your database. It also sorts the index keys, which can improve performance.

Wow … sleepy now. If you are in the market for an effective sedative, then I recommend reading the MySQL 5.0 Reference Manual.

SEC. 110. DAYLIGHT SAVINGS.
(a) AMENDMENT.—Section 3(a) of the Uniform Time Act of 1966 15 U.S.C. 260a(a)) is amended—
(1) by striking ‘‘first Sunday of April’’ and inserting ‘‘second sunday of March’’; and
(2) by striking ‘‘last Sunday of October’’ and inserting ‘‘first Sunday of November’’.
(b) EFFECTIVE DATE.—Subsection (a) shall take effect 1 year after the date of enactment of this Act or March 1, 2007, whichever is later.
(c) REPORT TO CONGRESS.—Not later than 9 months after the effective date stated in subsection (b), the Secretary shall report to Congress on the impact of this section on energy consumption in the United States.
(d) RIGHT TO REVERT.—Congress retains the right to revert the Daylight Saving Time back to the 2005 time schedules once the Department study is complete.

If these two economists are right, then there’s a good chance we get to do the last few weeks over again in order to put everything back.

#!/bin/bash

if [ -t 0 ]; then
        echo 'You are not allowed to login to this shell.'
else
        /bin/bash ${1} "${2}"
fi
exit 0

After making the script executable, it has the following results:

> scp filename.txt username@machine:
username@machine's password: filename.txt
100% 9695     9.5KB/s   00:00
> ssh username@machine
username@machine's password:
You are not allowed to login to this shell.
Connection to machine closed.

The reason that this works is that both scp and sftp work by initializing an ssh connection and starting scp or sftp on the server side. Basically, the command

scp filename username@machine:

is, initially at least, the same as sending the command

ssh username@machine ${SHELL} -c "scp -t ."

By changing the user’s shell to my script, I am effectively running that test “-t 0″ before giving them the opportunity to start the command that they are sending over. The test checks to see if the command is being run on a terminal or not.

Now, there are several issues with the approach that I took. The first is that a user who understands what I’m doing could very easily get around this. There’s nothing to stop them from sending their own commands instead of scp or sftp. This could be fixed rather easily though, all that someone would need to do is test ${2} to make certain that the string begins with either scp or sftp. The other issue that I have is that a user could easily write to files outside of their home directory. After testing to see if the command is scp or sftp all someone would have to do is replace ${2} with the appropriate command followed by a ” -t .” That would effectively close those two holes.

Here’s a perl version that I just found from the Snail Book. The author addresses the first issue that I have with my bash version in basically the same way I described above.