SSH using keys

September 5th, 2010 Behzad No comments

What are SSH Keys?

By using SSH Keys (a public and private key to be precise), you can easily connect to a server, or multiple servers, without having to enter your password for each system.

It is possible to setup your keys without a passphrase, however that is unwise as if anyone gets hold of your key they can use it. This guide describes how to setup your system so that passphrases are remembered securely.

Generating SSH Keys

The keys can then be generated by running the ssh-keygen command as a user:

# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
45:54:10:01:9d:ef:a3:34:a6:d9:f3:a2:41:e3:87:b7 root@localsquid

It will prompt you for a location (which you should leave as the default), however the passphrase is the important bit! I hopefully need not tell you the rules of a good passphrase?
Default key length for RSA is 2048 and is sufficient.

Copying the keys to the remote server

Now you have generated the keys you need to copy them to the remote server. By default, for OpenSSH, the public key needs to be concatenated into ~/.ssh/authorized_keys.

# scp ~/.ssh/id_rsa.pub root@192.168.1.100:

This copies the public key (id_rsa.pub) to your remote server via scp (note the : at the end of the server address). The file ends up in the home directory, but you can specify another path if you like.

Next up, on the remote server, you need to create the ~/.ssh directory if it doesn’t exist and concatenate the key authorized_keys file:

# ssh root@192.168.1.100
root@192.168.1.100's password:
# mkdir ~/.ssh
# cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
# rm ~/id_rsa.pub
# chmod 600 ~/.ssh/authorized_keys

The last two commands remove the public key from the server (which isn’t needed now), and sets the correct permissions on the authorized_keys file.

If you now disconnect from the server, and attempt to reconnect, you should be asked for the passphrase of the key (if any):

# ssh root@192.168.1.100
Enter passphrase for key '~/.ssh/id_rsa':

If you are unable to login with the key, double check the permissions on the authorized_keys file.
Also check the permissions on the ~/.ssh directory, which should have write permissions off for ‘group’ and ‘other’. Run the following command to disable ‘group’ and ‘other’ write permissions for the ~/.ssh directory:

# chmod go-w ~/.ssh

Categories: Linux | CentOS Tags:

Analyze of Squid’s access.log

September 4th, 2010 Behzad No comments

Squid cache result

The cache result code appears in the fourth column of Squid’s native access.log format. It indicates how the request was handled by Squid.

The following codes, beginning with TCP_, are for requests to the HTTP port (3128).

TCP_HIT
A valid copy of the requested object is in the cache. Squid does not forward the request.

TCP_MISS
The requested object is not in the cache.

TCP_REFRESH_HIT
The object is in the cache, but stale. Squid has forwarded an If-Modified-Since request and received a Not Modified response.

TCP_REF_FAIL_HIT
The object is in the cache, but stale. Squid has forwarded an If-Modified-Since request but it failed (e.g. connection timeout), so Squid sends the old (stale) object to the client.

TCP_REFRESH_MISS
The object is in the cache, but stale. Squid has forwarded an If-Modified-Since request and received a response with the new content.

TCP_CLIENT_REFRESH
The client issued a request with the no-cache pragma, so Squid forwards the request.

TCP_IMS_HIT
The client issued an If-Modified-Since request and the object is in the cache, and still fresh. Squid does not forward the request.

TCP_IMS_MISS
The client issued an If-Modified-Since request for a stale object. Squid forwards the request as a miss.

TCP_SWAPFAIL
The object is believed to be in the cache, but could not be accessed, so Squid forwards the request.

TCP_DENIED
Access is denied for this request.

The following codes, beginning with UDP_, are for requests to the ICP port (3130).

UDP_HIT
fresh copy of the requested object is in the cache.

UDP_HIT_OBJ
Same as UDP_HIT, but the object data is small enough to also be sent in the UDP reply packet.

UDP_MISS
The requested object is either stale or not in the cache at all.

UDP_DENIED
Access is denied for this request.

UDP_INVALID
An invalid request is received.

UDP_MISS_NOFETCH
The replying cache recommends that this request NOT be made at this time. Squid uses this reply code when reloading metadata at startup, and when the failure ratio exceeds its threshold.

There are also numerous error codes, which begin with ERR_, for requests to the HTTP port. We will not discuss these codes here.

Every HTTP response includes a status code on the first line. This three-digit numeric code also appears in the fourth column of Squid’s native access.log. Section 6.1.1 of the HTTP/1.1 RFCincludes a full list of these codes. We are primarily interested in only two of them.

200 OK
This, the most common status code, indicates a successful request.

304 Not Modified
This means the request included an If-Modified-Since header with a timestamp, and the resource has not been modified since the given time.

The ninth column of Squid’s native access.log is a code that indicates how the next-hop cache was selected.

DIRECT
Squid forwards the request directly to the origin server.

FIREWALL_IP_DIRECT
Squid forwards the request directly to the origin server because the origin server’s IP address is inside your firewall.

FIRST_PARENT_MISS
Squid forwards the request to the parent cache with the fastest weighted round trip time.

FIRST_UP_PARENT
Squid forwards the request to the first available parent in your list.

LOCAL_IP_DIRECT
Squid forwards the request directly to the origin server because the origin server’s IP address matched your local_ip list.

SIBLING_HIT
Squid forwards the request to a sibling cache that sent us a UDP_HIT reply.

NO_DIRECT_FAIL
Squid cannot forward the request because of firewall restrictions and no parent caches are available.

PARENT_HIT
Squid forwards the request to a parent cache that sent us a UDP_HIT reply.

SINGLE_PARENT
The request is forwarded to the only parent cache appropriate for this request. Also requires that single_parent_bypass be enabled.

SOURCE_FASTEST
Squid forwards the request to the origin server because its source_ping reply arrived first.

PARENT_UDP_HIT_OBJ
Squid received the object in a UDP_HIT_OBJ reply from a parent cache.

SIBLING_UDP_HIT_OBJ
Squid received the object in a UDP_HIT_OBJ reply from a sibling cache.

DEFAULT_PARENT
Squid forwarded the request to a default parent, without sending any ICP queries first.

ROUNDROBIN_PARENT
Squid forwarded the request to the round-robin parent with the lowest use count, without sending any ICP queries first.

CLOSEST_PARENT_MISS
Squid forwarded the request to the the parent whose ICP_MISS reply has the lowest measured RTT to the origin server. This only appears with query_icmp enabled in the configuration file.

CLOSEST_DIRECT
Squid forwarded the request directly to the origin server because Squid measured a lower RTT to the origin than any of its parent caches.

NONE
Squid does not forward the request at all.

Note, when the two second timeout occurs waiting for ICP replies, the word TIMEOUT_ is prepended to the hierarchy code

Categories: Squid Tags:

CentOS 5 Xen images – part 2: installing the Xen guest

September 2nd, 2010 Behzad No comments

OK, so you downloaded an image from jailtime.org or you made your own image as described in my previous post, and now you want to install a Xen guest using it. Here’s how to do it :

Setup the dom0 (host OS)

Of course, Xen should be installed on the host OS. I use CentOS 5, so I just selected Xen during the installation. It will install a xen kernel that you should use to boot the host OS (dom0). I won’t go into details here, because that’s really easy to do with CentOS or Redhat Enterprise Linux. With other distributions, you could have to install distribution-specific packages, or use the official Xen package fromxen.org.

Setup the target partition or logical volume

You should create a filesystem for the “root” partition and the swap. You could use simple files, but you will have better performance using real partitions or LVM volumes. LVM volumes also has other advantages, like the ability to create snapshots for backing up data, and easy resizing.

The following commands will create a 5GB root logical volume (LV) and 1GB swap in the /dev/vg0 volume group (VG). For more information about LVM, search for LVM howto in a search engine.

# root
lvcreate -L 5000M -n mailroot /dev/vg0
mkfs.ext3 /dev/vg0/mailroot
# swap
lvcreate -L 1000M -n mailswap /dev/vg0
mkswap /dev/vg0/mailswap

You can then mount the root partition and copy the base system (either an image from jailtime or an image you made yourself) on it.

mkdir /mnt/mailroot
mount /dev/vg0/mailroot /mnt/mailroot
# if you image contents is located in /centos...
cp -R /centos/* /mnt/mailroot/

Don’t forget to unmount the root partition when you’re done! Xen will not boot the domain if the partition is already mounted.

Download a kernel for the domU

The kernel that we will need to boot the domU has be to located in the dom0.

You can use the standard xen kernel that comes with CentOS to do that (e.g. vmlinuz-2.6.18-53.1.13.el5xen) , but you’ll also need an initrd, or the kernel won’t boot. To make the initrd, use the following command :

/sbin/mkinitrd --with=xennet --preload=xenblk /boot/initrd-centos5-xen.img 2.6.18-53.1.13.el5xen

This makes an initrd image with the required modules to boot a domU. The last parameter is the version of your kernel (the one you will use to boot the domU). You can get this number by typing “uname -r” on the command line. This will result in a /boot/initrd-centos5-xen.img image file.

Note (2008-02-14) : in a previous version of this blog post, I recommended to use a kernel from the official Xen distribution at xen.org. It worked, but it doesn’t seem to work anymore.

Create the configuration file

The configuration of the Xen guest is controlled by a simple text file. Create it as /etc/xen/yourdomUname, and move (or symlink) it in /etc/xen/auto if you want to start it automatically on boot.

Most basic parameters in this file are easy to understand. You should make sure “kernel” points to the kernel you copied from the xen tarball. “memory” is the amount of RAM allocated to the guest. “name” will be the name of the guest that you will use when connecting to it or shutting it down using the “xm” command.

“vif” contains information about network interfaces. One important thing in that line is the MAC address. If you don’t specify it here, a random MAC will be assigned at each boot, and that may not give good results. Edit the last 3 numbers (put anything, it just has to be unique across your network).

Finally, ‘disk’ is the parameter that tells Xen what partitions to use and what device name it will assign them. The last line, ‘root’, will tell the kernel what is the root device.

kernel = "/boot/vmlinuz-2.6.18-53.1.13.el5xen"
ramdisk = "/boot/initrd-centos5-xen.img"
memory = 512
name = "mail"
vif = [ 'mac=00:16:3e:21:f1:31,bridge=xenbr0' ]
dhcp = "dhcp"
disk = ['phy:/dev/vg0/mailroot,sda1,w', 'phy:/dev/vg0/mailswap,sda2,w' ]
# The next line would be useful if you want to use an simple file instead of a partition/LV
#disk = ['file:/root/test.img,sda1,w', 'file:/root/centos.swap,sda2,w' ]
# We don't use pygrub, we boot the kernel directly from dom0
#bootloader="/usr/bin/pygrub"
root = "/dev/sda1 ro"

Boot the domain!

OK, you’re ready to boot the guest domain! Just issue the following command to “create” (which means boot, really) the domU.

xm create /etc/xen/YOUR_CONFIG_FILE -c

The -c parameter tells xm to connect to the domain’s console. You can disconnect from it by pressing CTRL+], and connect to it again with “xm connect NAME”.

If everything works right, you should see the login prompt appearing, and you will be ready to use the new guest domain!

Fix SSH

If you made the guest image yourself as I explained in my previous post (part 1), you need to create the random device to fix SSH (and probably other services that requires generating keys). Issue the following commands on the guest’s console :

/sbin/MAKEDEV generic
/etc/init.d/sshd start

CentOS 5 Xen images – part 1: creating a base domU image with yum

September 2nd, 2010 Behzad No comments

I recently needed to configure Xen guests on a CentOS server. I didn’t want to use CentOS/Redhat’s tool for several reasons : I don’t like the fact that it creates a “disk” with a whole partition table within the partition where you install it. Installing the guest directly on a LVM logical volume seems better to me (I may not be right, I’m still somewhat new to Xen!). Also, it was very instructive to do all the steps by myself rather than using a tool that does everything.

However, I had several problems while trying to install my first guest system, so it took me quite some time to find the right way to do it. I found a lot of documentation on the web, but nothing really described what I really wanted to do. So here we are, I decided to write a post describing what to do exactly to make a custom guest image and install it as a Xen domU.

In this first part, I will explain how to make an OS image of CentOS 5 that will be suited for Xen.

Ready to use images

Jailtime.org offer Xen images for several operating systems, including CentOS 4 and 5. However, only 32-bit images are available. If 32-bit images are OK for you, I recommend that you use the jailtime images : it will be a lot easier.
A note about the jailtime.org CentOS image : I found that the console was not working on boot. The OS seemed to just hang after starting SSHD, but in fact, it was just that there was no console for Xen to connect to. If you have this problem, see the part about configuring the console in the “Custom CentOS 5 image” section of this article.

Custom CentOS 5 image

Since I wanted 64-bit domUs, I decided to make my own images. There are few things that need to be modified to make a standard CentOS installation work as a Xen domU. Here are the steps needed to make your own image with yum.

Prepare the environment

  1. Prepare a directory where you will store your image. I will use /mnt/centos.
    Note that you may want to chroot into the image when you need to work within it only. To do so, type :
    cd /mnt/centos; chroot .
    (Don’t chroot now, since there’s nothing in there for now)
  2. Create a special yum configuration file. We need a yum.conf file with “hard coded” architecture and version, instead of the variables the default yum.conf file contains.
    Create /etc/yum-xen.conf with the following content.

    [main]
    cachedir=/var/cache/yum
    keepcache=1
    debuglevel=2
    logfile=/var/log/yum.log
    pkgpolicy=newest
    distroverpkg=redhat-release
    tolerant=1
    exactarch=1
    obsoletes=1
    gpgcheck=1
    plugins=1
    metadata_expire=1800[base]
    name=CentOS-$releasever - Base
    mirrorlist=http://mirrorlist.centos.org/?release=5&arch=x86_64&repo=os
    gpgcheck=1
    gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
    #released updates
    [updates]
    name=CentOS-$releasever - Updates
    mirrorlist=http://mirrorlist.centos.org/?release=5&arch=x86_64&repo=updates
    gpgcheck=1
    gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
    #packages used/produced in the build but not released
    [addons]
    name=CentOS-$releasever - Addons
    mirrorlist=http://mirrorlist.centos.org/?release=5&arch=x86_64&repo=addons
    gpgcheck=1
    gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
  3. Install the core packages.
    We will use yum to install a core CentOS system in our target directory.

    yum -c /etc/yum-xen.conf --installroot=/centos -y groupinstall core --disablerepo=extras

    Here, we disable the “extras” repository since we don’t need it, and the variables it contains will make yum fail. You may need to disabled other repositories if you have installed custom ones.

You should now have a base system in your /centos directory ! But that’s not all, we need to modify few things to make our system suitable as a Xen guest.

Changes needed to the base system

As said earlier, you should now chroot in the /centos directory for the next steps. If you don’t do that, you could end up modifying your host OS instead of the image.

  1. Console
    You need to add a console to /etc/inittab, or Xen won’t display any login prompt when the image boot. You can also remove the default TTYs (tty1-tty6), since they won’t be needed in a Xen guest.

    # add a console for xen
    co:2345:respawn:/sbin/mingetty console
    # comment out the default TTYs
    #1:2345:respawn:/sbin/mingetty tty1
    #2:2345:respawn:/sbin/mingetty tty2
    #3:2345:respawn:/sbin/mingetty tty3
    #4:2345:respawn:/sbin/mingetty tty4
    #5:2345:respawn:/sbin/mingetty tty5
    #6:2345:respawn:/sbin/mingetty tty6
  2. You need to disabled TLS (it would slow down the guest)
    # for 32 bit :
    mv /lib/tls /lib/tls.disabled
    # for 64 bit :
    mv /lib64/tls /lib64/tls.disabled
  3. Disable the hardware clock
    The Xen domU will use the host’s clock, so we replace hwclock with an empty shell script

    echo exit 0 > /sbin/hwclock
  4. Disable udev in /etc/rc.sysinit
    Apparently udev is problematic with Xen, so we will comment the line where it’s started in rc.sysinit (line 338):

    #/sbin/start_udev

    UPDATE 2008-02-14 : It seems that it may be a good idea to leave udev enabled after all… I’m getting permissions problem on devices such as /dev/null without udev. Enabling it fix the problem, and does not seem to cause any other problem.

  5. Network configuration.
    If you want, you can define network parameters.
    /etc/sysconfig/network-scripts/ifcfg-eth0

    TYPE=Ethernet
    DEVICE=eth0
    BOOTPROTO=static
    BROADCAST=192.168.0.255
    IPADDR=192.168.0.13
    IPV6ADDR=
    IPV6PREFIX=
    NETMASK=255.255.255.0
    NETWORK=192.168.0.0
    ONBOOT=yes

    /etc/sysconfig/network

    NETWORKING=yes
    HOSTNAME=myhost.l3i.ca
    GATEWAY=192.168.0.1

    /etc/resolv.conf (DNS resolver)

    nameserver 4.2.2.1
    nameserver 4.2.2.2
  6. SELinux
    You may want to disabled SELinux. That’s done in /etc/sysconfig/selinux (put SELINUX=disabled)
  7. Root password
    We need to create a root password and create the password database (or we wouldn’t be able to log in the system).

    pwconv
    passwd root
  8. Create the /etc/fstab file.
    In this file, /dev/sda1 is the root (ext3) filesystem and sda2 is swap. You will need to configure Xen with these device names.

    cat > /etc/fstab
    # This file is edited by fstab-sync - see 'man fstab-sync' for details
    /dev/sda1               /                       ext3    defaults 1 1
    /dev/sda2               none                    swap    sw       0 0
    none                    /dev/pts                devpts  gid=5,mode=620 0 0
    none                    /dev/shm                tmpfs   defaults 0 0
    none                    /proc                   proc    defaults 0 0
    none                    /sys                    sysfs   defaults 0 0
    #(press CTRL+D to end "cat")
  9. Disable HAL
    HAL will fail, so let’s just disable it.

    /sbin/chkconfig haldaemon off
  10. SSH
    A note about SSH : it will fail to generate a key on the first load because there is not /dev/urandom device. You will have to fix it after starting the domU, on the console. The following command will fix it :

    /sbin/MAKEDEV generic
    /etc/init.d/sshd start

You now have an image ready to be used as a Xen guest! Be sure to keep a copy before using it as a domU, so when you need to install another guest, you will have an image ready for it and won’t need to repeat these steps all over again.

In part 2, I will explain how to install a Xen guest from this image. (coming soon!)

/vz partition errors or when /vz goes read-only

September 2nd, 2010 Behzad No comments

Note : Be Careful while using these steps.

A few times now we have seen problems on the Linux virtuozzo servers where the /vz partition errors out and the kernel forces it to go into a read-only state. The solution is to manually run a filesystem check (FSCK) with the -fy arguments on the /vz partition. It is possible on a vz server to do this without a support ticket, however you must be careful with a few steps.

1. Stop the vz service (service vz stop)

2. Disable the vz service from restarting on boot (chkconfig –level 2345 vz off)

3. Comment out the /vz partition inside the /etc/fstab file to prevent the /vz partition being automaticlyt checked on boot.

4. Reboot the server

5. Once the server reboots and you can SSH back in you can begin the filesystem check.

6. Determine which physical partition is /vz/ You can do this with the command (fdisk -l /dev/sda). This will show you all the partitions.

Usually the /vz partition is the last one listed and will have the larges amount of blocks with a “ID” of 83 and “System” label of Linux:

EXAMPLE
# fdisk -l /dev/sda

Disk /dev/sda: 299.9 GB, 299978719232 bytes
255 heads, 63 sectors/track, 36470 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1305 10482381 83 Linux
/dev/sda2 1306 2610 10482412+ 83 Linux
/dev/sda3 2611 3915 10482412+ 83 Linux
/dev/sda4 3916 36470 261498037+ 5 Extended
/dev/sda5 3916 4437 4192933+ 82 Linux swap
/dev/sda6 4438 4568 1052226 83 Linux
/dev/sda7 4569 36470 256252783+ 83 Linux

In the above example you can see that /dev/sda7 have the most blocks and has the “ID” of 83 and “System” label of Linux:.

Therefore we would issue the command (fsck -fy /dev/sda7).

7. Once the file system check is complete you can uncomment the /vz parition in /etc/fstab

8. re-enable the vz service to start on boot (chkconfig –level 2345 vz on)

9. start up the vz service (service vz start)

All should then be well. You can monitor the logfile /var/log/vzctl.log as VE’s are brought online.

If you are uncomfortable doing any of this then please contact a senior tech or management and we can help out. Please remember it is very dangerous to FSCK a mounted partition to always ensure the target partition os unmounted before beginning. Generally you cannot umout a /vz partition after stopping the VZ service so I recommend the reboot with /vz partition disabled and the vz service disabled.

Categories: Linux | CentOS, OpenVZ Tags: