Run

Top 10 memory consumers
ps auxf | sort -nr -k 4 | head -10

Top 10 CPU consumers
ps auxf | sort -nr -k 3 | head -10

I have recently setup a master and slave in MySQL.

the main reason I did this is so I could do some queries and light data mining looking for patterns in our datasets. The issue is that the Live database is optimized for inserts and updates, not for views or summaries.

in creating a replicated database, I was able to index my slave differently, and allow it to be better suited for pulling this report views.

I am NO MySQL database expert, but was able to follow the simple directions on the mysql.com website and have this up and running relatively quickly and painlessly.

This is how I did it:

  • I configured my master by adding the following to my.cnf and restarting the mysql instance:
  • server-id=1276024641
    log-bin=mysql-bin
    log-error=mysql-bin.err
    binlog_do_db=<my_target_db_name>
  • the db_name restricts only 1 db to be replicated. I think omitting this line will replicate all the databases on the master to the slave
  • I added a unique, random number as my server-id to my slave machine’s my.cnf file. the line I added looked like this : server-id=1281421047
  • I then restarted the slave mysql instance
  • I then setup the replication user on the master. I used the phpMyAdmin scripts, by clicking on the “Replication” tab, then the “add slave replication user” link. I saved this info
  • because I am running a live database, I needed to locate the master’s bin log coordinates. To do this, you need to ensure no one is changing the data, the manual recommends locking the tables, but on our transaction heavy system, I just closed the db for the next step.
  • I then proceeeded to output a mysqldump: mysqldump -p my_database_name > db_script.sql
  • while this was saving to file, I went into phpmyadmin under replication and clicked “show master status”, which displayed
  • Variable     Value
    File                mysql-bin.000120
    Position       83336594
    Binlog_Do_DB    my_databaswe_name
    Binlog_Ignore_DB
  • Once the data dump was complete, I re-enabled traffic back into the master db.
  • I then copied the .sql file to the slave machine and imported it using: mysql -p my_database_name < db_script.sql
  • Once the db snapshot was imported, I needed to tell the slave to start running the slave process thread, by running the query:
  • CHANGE MASTER TO
    MASTER_HOST=’master_host_name’,
    MASTER_USER=’replication_user_name’,
    MASTER_PASSWORD=’replication_password’,
    MASTER_LOG_FILE=’recorded_log_file_name’,
    MASTER_LOG_POS=recorded_log_position;
  • from there, the slave should keep in sync with the master

I’ve done this a few times setting up MRTG and always get stuck when trying to use custom scripts.

The caveat is when you target a custom script the notation is:

Target[localhost.mem]: `/etc/mrtg/scripts/mem.pl`

or something along those lines, the caveat is that the quotes are not a ‘ but a ` ( the character at the top left of the keyboard)   this took me a little while to figure it out.

The perl script I am using is:

#!/usr/bin/perl

$machine = `/bin/hostname`;
chomp($machine);
$mem = `/usr/bin/free | grep Mem`;
$uptime = `/usr/bin/uptime`;

if ($mem =~ /^Mem:\s*(\d*)\s*(\d*)\s*(\d*)/) {
 $tot = $1;
 $used = $2;
 $free = $3;
}

if ($uptime =~ /up (.*),  \d* users?,/) {
 $up = $1;
}
print "$used\n";
print "$free\n";
print "$up\n";
print "$machine\n";

I took this from another site.. I will probablly write my shell scripts in PHP, they are just as easy , and powerful

So finally got a chance to get on the Amazon EC2 hosting cloud. Got myself a small “instance” a virtual machine equivalent to a 1 gig xeon. Hmm instead blogging about it, I should probablly get my web applications moved over to it.

So I will do that, then update this post with more info. So far the Amazon system is pretty awesome, it’s fast, I was able to shutdown and start about 7 instances in about a 1 hr timeframe trying out different starter images.

I ended up settling on a base Debian 5.0 Lenny , and re-setup all my services from scratch. This takes time but ensures I know what services I have running and forces me to re-look into the configurations I setup. Fortunately I can do cool things like copy the entire /var/lib/mysql folder over and all my databases move no problems :). The bandwidth on the amazon instances is capped about 10 Mbit/s which is pretty decent.. I’ve seen my transfers average 8 Mbit/s easy..

So better get back to it, and move another domain over.

I have had to do this a few times, and it’s really handy. Making a SSH keypair that will allow me to login on a remote machine without entering a password.

Here are the commands:

On my machine (primary host) I want given permission to access remote machines

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ryan/.ssh/id_rsa): <ENTER>
Enter passphrase (empty for no passphrase):  <ENTER>
Enter same passphrase again:  <ENTER>
Your identification has been saved in /home/ryan/.ssh/id_rsa.
Your public key has been saved in /home/ryan/.ssh/id_rsa.pub.
The key fingerprint is:
fa:e7:7c:e1:cb:7b:66:8b:67:07:05:99:7f:05:b9:4a ryan@localhost

I then copy id_rsa.pub out to the remote machines user directory, so my localhost user can access the remote host user account without password, a network of trusted hosts .

$ mkdir remote_server/.ssh
$ chmod 700 remote_server/.ssh
$ cat .ssh/id_rsa.pub >> remote_server/.ssh/authorized_keys
$ chmod 600 remote_server/.ssh/authorized_keys

I’ve had to do this task a good many times, soI figured I should note it here for easy access.

Problem: How do I format an external USB drive for access in the linux filesytem.

First plug in the drive and power it up.
To ensure the Linux OS recognizes the new device, you can run the command

dmesg

It should output something like:

 I/O error: dev 08:10, sector 0
 unable to read partition table
 I/O error: dev 08:10, sector 0
Device 08:20 not ready.
 I/O error: dev 08:20, sector 0
SCSI device sda: 625142447 512-byte hdwr sectors (320073 MB)
 /dev/scsi/host2/bus0/target0/lun0: p1

What you are looking for now is the device the OS has mapped it to.. in this case it’s sda ( it may also be sdb or sdc, depending on what other devices you have connected )

You will now be able to access this device through /dev/sda
Next I ran

cfdisk /dev/sda

This allows me to view the partition table on this device. It should show you the type of filesystem if any ( ntfs,fat or unformatted, etc )

In my case I want to make it a linux partition.
I created a [new partition], and set the type to 83 – Linux

I then chose [write] and [quit], this will create the new partition ( and WARNING!! :permanently erase everything on the device)
I could then format the device. I use ReiserFS, which I find great for devices that get jostled around and may get slight corruption, Reiser is awesome at recovering and fixing itself, even better than Ext3
I ran

mkreiserfs /dev/sda1

The 1 on sda1 now signifies the first partition on the device sda.
This will format the device and let you know the result.

At this point I did a quick check to see the health of the device with the command

fsck.reiserfs –check /dev/sda1

Which output:

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Tue Feb 10 09:57:44 2009
###########
Replaying journal..
Reiserfs journal '/dev/sda1' in blocks [18..8211]: 0 transactions replayed
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
        Leaves 1
        Internal nodes 0
        Directories 1
        Other files 0
        Data block pointers 0 (0 of them are zero)
        Safe links 0
###########
reiserfsck finished at Tue Feb 10 09:58:06 2009
###########

the key info there is the line : No corruptions found
Which means there are no Bad blocks on the device.
So now I proceed to mount the device and then access it like any other folder on the system
mount -t auto /dev/sda1 /mnt/BACKUP-DRIVE/
in this case I had already made a folder called BACKUP-DRIVE in my /mnt directory.
now If I run:


df

I get

/dev/sda1            312559096     32840 312526256   1% /mnt/BACKUP-DRIVE

That tells me the drive is ready to go, I can cd /mnt//BACKUP-DRIVE and start writing files.
it my case I was to rsync my work files into the backup device I use this command


rsync -vbruz /work /mnt/BACKUP-DRIVE

This will check any only update the files that have changed since the last backup, and go recursively.. in this case, it will copy everything over and may take awhile.
When this is complete.. I run
umount /dev/sda1

and I can safely unplug the device until the next backup.
For the next backup I simply

  1. Plug device in and turn on
  2. mount -auto /dev/sda1 /mnt/BACKUP-DRIVE
  3. rsync -vbruz /work  /mnt/BACKUP-DRIVE ( where /work is the source of the files I want to backup )

ReiserFs comes with a whole bunch of great tools, in case the drive does become corrupted.. or it’s starts to age and gets bad blocks.

You can also automate alot of this ( I have yet to do this ) so that when you plugin the device it automatically calls your backup scripts or something.