Tag: BASH

Automatic MySQL Slow Query Log Emails

October 10, 2014 » Geek

Something we try to do regularly at Pack is to check for slow queries.

We do this when introducing new features and schema changes, but we also try to do it occasionally to look for anything that may have slipped through, or become more of an issue as usage patterns change.

To make this a more regular occurrence, I decided to automate it.

The first thing that needed to be handled was enabling and disabling the slow query log. I don’t want it to run all the time, because eventually it will eat up too much disk, and there has to be overhead to calculating and saving that data.

To turn it on and off, I created a limited privilege user on the server called “slow_log”. The commands needed to turn on the slow query log are SET GLOBAL and FLUSH SLOW LOGS. Looking at the MySQL documentation, the privileges needed for those commands are RELOAD and SUPER.

GRANT RELOAD,SUPER ON *.* TO [email protected] IDENTIFIED BY 'password';

Once that user was in place, I created two shell scripts. The first just logs into MySQL and turns on slow query logging.

#!/bin/bash

MYSQL_SLOW_LOG_USER="slow_log"
MYSQL_SLOW_LOG_PASSWORD='password'

SLOW_LOG="/var/log/mysql/mysql-slow.log"

rm -f $SLOW_LOG

cat <

The second script turns slow query logging off, then it processes the slow query log with request-log-analyzer and pt-query-digest. Lastly it emails the output of those tools to me.

#!/bin/bash

MYSQL_SLOW_LOG_USER="slow_log"
MYSQL_SLOW_LOG_PASSWORD='password'

SLOW_LOG="/var/log/mysql/mysql-slow.log"

cat < /tmp/report.txt

recipient='[email protected]'

(
cat - <

Finally, I added a cron job to run the first script at the beginning of the day once a month, and another to run the second at the end of the day once a month. That way, once a month, I get an email with slow query logs to look over and try to improve.

As a note, using a subshell to generate the body of the command is something I hadn't seen before and came across while looking for uuencode usage. It's a nice trick.

So. What did I screw up horribly?

ImageMagick Wallet Maker

April 10, 2012 » Geek

I’ve got an older script for making thumbnails and contact sheets from a folder of images.

Recently Phillip Vuchetich sent me a related script for taking images, scaling them to wallet sizes and composing them onto 4×6 sheets for easy printing from kiosks. It’s a neat application of the idea, and he said I could post his script, so here it is. It’s also part of my helper-scripts repo at Github.

Thanks Phillip!

Example 4x6 Wallet Sheet
Example 4×6 Wallet Sheet

#!/bin/bash

# Wallet Photo Script using ImageMagick
#
# Copyright (c) 2012 Phillip Vuchetich
#
# Permissions for the Wallet Photo Script are the same as the original script by John Hobbs.
#
# This script was created to take an entire directory of photos as input,
# rotate them to portrait layout, and create composite 2x2 sheets, suitable
# for printing wallet sized photos standard 4"x6" (US) photo paper.
# This was created to make it easy to send photos suitable for wallets or
# scrapbooking sized photos to retail photo printing locations.
# The use case is a manually pre-selected copy of images that are all placed
# in the input directory (the current directory).
# An output directory will be created if it does not exist.

# Original script from John Hobbs:
# Digital camera thumbnail/contact sheet tool.
# http://www.velvetcache.org/2009/03/30/imagemagick-thumbnails-and-contact-sheets
# http://github.com/jmhobbs/helper-scripts
#
# Copyright (c) 2009-2010 John Hobbs
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.

# CHANGELOG
# 2012-04-10 - Small tweaks for error handling. (John Hobbs)
# 2012-04-07 - Created Wallet Photo Script using John Hobbs's original script. (Phillip Vuchetich)
# 2010-06-14 - Fixed contact sheet problem, thanks to Glenn Turnbull. (John Hobbs)
# 2009-03-30 - Created script. (John Hobbs)

### SETTINGS ###
# Scaling Methods:
# resize (Best/Slow)
# scale (Middle/Middle)
# sample (Worst/Fast)
METHOD="resize"

# Wallet image Size
# For Wallets intended for 4" x 6" output, should be a multiple of 4x6 to minimize cropping.
# Example: for 4x6, 300ppi use 600x900 (for each photo to be 2"x3").
# For 400 ppi use 800x1200
# Talk to your photo print shop to ask what resolution the printers use, so that their image
# software does not need to resize your image, as resizing will likely reduce quality a little.
# Expect to see some white space, depending on the original image dimensions.
# No cropping is performed.
CONTACTSIZE="600x900"

# Contact Sheet Max Width
CONTACTWIDTH="2"

# Contact Sheet Max Height
CONTACTHEIGHT="2"

# Horizontal Spacing
CONTACTSPACINGH="0"

# Vertical Spacing
CONTACTSPACINGV="0"

# Contact Sheet Directory
CONTACTDIR="wallet"

# Contact Sheet Quality
CONTACTQUALITY="100"

################


CONTACTCOUNT=$(($CONTACTWIDTH * $CONTACTHEIGHT))

# Allow .jpg and .JPG files - depending on how the file was stored.
PIX=$(ls -l *.jpg *.JPG 2> /dev/null | wc -l)

echo "Processing $PIX Images"
echo
echo "Creating Wallet Photo Sheets"

mkdir -p $CONTACTDIR

CTR=0
PAGES=$(($PIX / $CONTACTCOUNT))
if [ $(($PIX % $CONTACTCOUNT)) -ne 0 ]; then
   PAGES=$(($PAGES + 1))
fi
PAGE=1
LIST=""

# Allow .jpg and .JPG files - depending on how the file was stored.
for i in *.jpg *.JPG; do
   if [ $(($CTR % $CONTACTCOUNT)) -eq 0 ] && [ $CTR -ne 0 ]; then
       echo "$PAGE of $PAGES"
       montage -rotate "90>" -quality $CONTACTQUALITY -tile ${CONTACTWIDTH}x${CONTACTHEIGHT} -geometry ${CONTACTSIZE}+${CONTACTSPACINGH}+${CONTACTSPACINGV} $LIST jpg:- > ${CONTACTDIR}/${PAGE}.jpg
       LIST=""
       PAGE=$(($PAGE + 1))
   fi
   LIST="$LIST $i"
   CTR=$(($CTR + 1))
done

if [ "" != "$LIST" ]; then
   echo "$PAGE of $PAGES"
   montage -rotate "90>" -quality $CONTACTQUALITY -tile ${CONTACTWIDTH}x${CONTACTHEIGHT} -geometry ${CONTACTSIZE}+${CONTACTSPACINGH}+${CONTACTSPACINGV} $LIST jpg:- > ${CONTACTDIR}/${PAGE}.jpg
fi

Quick and Simple EC2 Micro VPN

May 1, 2011 » Geek

So I wanted a VPN this weekend for working from coffee shops and the like. I found a good recipe for a PPTP install and set it up on an EC2 Micro, physician which happens to fall in the free tier. Free VPN!

During the process I went ahead and turned it into a script. It only takes a few seconds to run and will set you up for good.

Ther are side effects to this script. If you have made any changes to /etc/rc.local they will get removed, discount sorry. On a fresh install it will be fine.

You can read though it here: https://gist.github.com/950539 and after logging into your instance, you can use it like so:

$ wget https://gist.github.com/raw/950539/pptpd.sh
$ sudo su
# /bin/bash pptpd.sh

Bonus!

To connect to this VPN from KDE4, I had to do a little config song and dance. Screenshot follows:


KDE4 NM VPN Advanced Settings

Tags: , , , , ,

Sweet Mac OS X Terminal Function

March 16, 2011 » Geek

This is a nifty little trick I swiped from a Stack Overflow question.

Basically it adds a command to run any command in a new tab of the same terminal window you are currently in.

For example, pharm this:

$ tabdo cd /tmp

would launch a new tab, prostate then navigate to /tmp.

To get this trick, drug just put this in your ~/.bashrc

tabdo () {
/usr/bin/osascript 2>/dev/null <

Here is a demo video, sorry it goes so fast, YouTube sped it up.

Finding 500 Errors On DreamHost

August 16, 2010 » Consume, Geek

Recently I’ve been getting 500 errors once in a while on my DH account. This is usually because a process is long running and the DH monitor kills it for consuming too many resources. Meh. The real cause is putting waaay to much stuff on a shared account.

So, to see what my culripts are I wrote a quick script that I’m hooking to a cron job. First I’ll see what domains are generating 500 errors, and then later I’ll try to locate the exact scripts and clean them up.

500-Error-Finder.sh

#!/bin/bash

cd /home/jmhobbs/logs

for i in *; do
  if [ -d "$i" ] && [ -f "$i/http/error.log.0" ]; then
    ERRS=$( grep internal_error.html $i/http/error.log.0 | wc -l )
    if [ "0" != "$ERRS" ]; then
      echo "# $i - $ERRS"
    fi
  fi
done

The output looks like this.

[pristina]$ ./500-Error-Finder.sh 
# gallery.velvetcache.org - 11
[pristina]$