defenestrations & lapidations

SFTP with Unicode support in proftpd

2011-08-22T19:56:00.001+08:00

Debian sid’s proftpd (1.3.4) still seems to have iffy Unicode support, so one way to get UTF8 working reliably in proftpd is to use an SFTP connection. The first thing to do is ensure a UTF8 locale is in use:

$ locale
LANG=en_GB.utf8

If you do not see “.utf8” appended to the LANG shell variable, execute the following:

# export LC_ALL=en_GB.utf8
# export LANG=en_GB.utf8

To ensure this is applied at every boot, add it to /etc/profile.

The rest is easy. Users with an ssh account and a defined shell in /etc/passwd already have SFTP access once proftpd is started. But what about non-shell accounts (i.e. FTP-only users)? /bin/false certainly won’t give SFTP access, so we use scponly instead. From the man page:

scponly is an alternative "shell" (of sorts) for system administrators who would like to provide access to remote users to both read and write local files without providing any remote execution privileges. Functionally, it is best described as a wrapper to the trusted suite of ssh applications.

To change a user’s shell to scponly:

# usermod -s /usr/bin/scponly username

It’s that easy.

Smbclient to NFS

2011-04-13T01:21:00.001+08:00

So I finally switched from smbclient to NFS. For the longest time smbclient had been giving me Gigabit-like writes (~70MB/s), but Fast-Ethernet-like reads (7MB/s), and I could never figure out why.

Today, with some Wiresharking, I finally narrowed down the issue: READ ANDX packet sizes being limited to 4096 bytes for any samba transfer between my home server and desktop (both running on Arch, kernel 2.6.38, smbclient 3.5.8-2). No amount of tweaking for any option, including but not limited to SO_*BUF, succeeded in changing the negotiated packet size to anything higher than 4096.

The last time I used NFS, it gave really bursty transfer performance, which left me rather displeased. The reason for that is now determined to be a sub-optimal mdadm/disk configuration (see previous post). With raid performance closer to what it should be, I am getting more consistent NFS transfer performance: 70MB/s reads, 100MB/s writes. A bug in the r8169 realtek driver module (firmware rtl_nic/rtl8168d-1, rtl_nic/rtl8168d-2) left me with a system that crashed on any network file transfer, so I used the r8168 driver instead.

# mv /lib/modules/2.6.38-ARCH/kernel/drivers/net/r8169.ko.gz /lib/modules/2.6.38-ARCH/kernel/drivers/net/r8169.ko.gz.bk
# yaourt -S r8168
# modprobe -r r8169; modprobe r8168
# depmod -a
# mkinitcpio -p kernel26

Check that eth0 is using the correct driver:

# ethtool -i eth0

Then add r8168 to the MODULES() list in /etc/rc.conf.

Linux disk performance tweaking parameters

2011-04-12T13:45:00.003+08:00

A quick reference and note to self. Will be fleshed out in greater detail in future.

NCQ queue depth: /sys/block/sd*/device/queue_depth
readahead blocks: /sbin/blockdev --getra /dev/md* (or /dev/sd*)
stripe cache size: /sys/block/md*/md/stripe_cache_size

[edit] A fuller version has been posted on Ubuntu forums. The tuneraid script I use is as follows:

#!/bin/bash

###############################################################################

#  simple script to set some parameters to increase performance on a mdadm

# raid5 or raid6. Ajust the ## parameters ##-section to your system!

#

#  WARNING: depending on stripesize and the number of devices the array might

# use QUITE a lot of memory after optimization!

#

#  27may2010 by Alexander Peganz

#  31jul211 modified by Mathias B

###############################################################################





## parameters ##

MDDEV=md127			# e.g. md51 for /dev/md51

CHUNKSIZE=512		# in KB

BLOCKSIZE=4			# of file system in KB

NCQ=enable			# disable, enable. ath. else keeps current setting

NCQDEPTH=31			# 31 should work for almost anyone

FORCECHUNKSIZE=true	# force max sectors kb to chunk size > 512

DOTUNEFS=true		# run tune2fs, ONLY SET TO true IF YOU USE EXT[34]

RAIDLEVEL=raid5		# raid5, raid6

logfile="/home/kureshii/tuneraid.log"



## code ##

# test for priviledges

if [ "$(whoami)" != 'root' ]; then

	echo $(date): Need to be root >> "$logfile"

	exit 1

fi



# set number of parity devices

NUMPARITY=1

[[ $RAIDLEVEL == "raid6" ]] && NUMPARITY=2

echo "Using $NUMPARITY parity devices ($RAIDLEVEL)"

# get all devices

DEVSTR="`grep \"^$MDDEV : \" /proc/mdstat` eol"

while \

 [ -z "`expr match \"$DEVSTR\" '\(\> "$logfile"

	exit 1

fi



echo "Found $NUMDEVS devices with $NUMSPAREDEVS spares"



# set read ahead

RASIZE=$[$NUMDEVS*($NUMDEVS-$NUMPARITY)*2*$CHUNKSIZE]   # in 512b blocks

echo read ahead size per device: $RASIZE blocks \($[$RASIZE/2]kb\)

MDRASIZE=$[$RASIZE*$NUMDEVS]

echo read ahead size of array: $MDRASIZE blocks \($[$MDRASIZE/2]kb\)

blockdev --setra $RASIZE /dev/sd[$DEVS]

#blockdev --setra $RASIZE /dev/sd[$SPAREDEVS]

blockdev --setra $MDRASIZE /dev/$MDDEV



# set stripe cache size

STRCACHESIZE=$[$RASIZE/8]                               # in pages per device

echo stripe cache size of devices: $STRCACHESIZE pages \($[$STRCACHESIZE*4]kb\)

echo $STRCACHESIZE > /sys/block/$MDDEV/md/stripe_cache_size



# set max sectors kb

DEVINDEX=0

MINMAXHWSECKB=$(cat /sys/block/sd${DEVS:0:1}/queue/max_hw_sectors_kb)

until [ $DEVINDEX -ge $NUMDEVS ]; do

	DEVLETTER=${DEVS:$DEVINDEX:1}

	MAXHWSECKB=$(cat /sys/block/sd$DEVLETTER/queue/max_hw_sectors_kb)

	if [ $MAXHWSECKB -lt $MINMAXHWSECKB ]; then

		MINMAXHWSECKB=$MAXHWSECKB

	fi

	DEVINDEX=$[$DEVINDEX+1]

done

if [ $CHUNKSIZE -le $MINMAXHWSECKB ] && ( [ $CHUNKSIZE -le 512 ] || [[ $FORCECHUNKSIZE == "true" ]] ); then

	echo Setting max sectors KB to match chunk size

	DEVINDEX=0

	until [ $DEVINDEX -ge $NUMDEVS ]; do

		DEVLETTER=${DEVS:$DEVINDEX:1}

		echo "Set max sectors KB to $CHUNKSIZE on $DEVLETTER"

		echo $CHUNKSIZE > /sys/block/sd$DEVLETTER/queue/max_sectors_kb

		DEVINDEX=$[$DEVINDEX+1]

	done

	DEVINDEX=0

	until [ $DEVINDEX -ge $NUMSPAREDEVS ]; do

		DEVLETTER=${SPAREDEVS:$DEVINDEX:1}

		echo "Set max sectors KB to $CHUNKSIZE on $DEVLETTER"

		echo $CHUNKSIZE > /sys/block/sd$DEVLETTER/queue/max_sectors_kb

		DEVINDEX=$[$DEVINDEX+1]

	done

fi



# enable/disable NCQ

DEVINDEX=0

if [[ $NCQ == "enable" ]] || [[ $NCQ == "disable" ]]; then

	if [[ $NCQ == "disable" ]]; then

		NCQDEPTH=1

	fi

	until [ $DEVINDEX -ge $NUMDEVS ]; do

		DEVLETTER=${DEVS:$DEVINDEX:1}

		echo Setting NCQ queue depth to $NCQDEPTH on $DEVLETTER

		echo $NCQDEPTH > /sys/block/sd$DEVLETTER/device/queue_depth

		DEVINDEX=$[$DEVINDEX+1]

	done

	DEVINDEX=0

	until [ $DEVINDEX -ge $NUMSPAREDEVS ]; do

		DEVLETTER=${SPAREDEVS:$DEVINDEX:1}

		echo Setting NCQ queue depth to $NCQDEPTH on $DEVLETTER

		echo $NCQDEPTH > /sys/block/sd$DEVLETTER/device/queue_depth

		DEVINDEX=$[$DEVINDEX+1]

	done

fi



# tune2fs

if [[ $DOTUNEFS == "true" ]]; then

	STRIDE=$[$CHUNKSIZE/$BLOCKSIZE]

	STRWIDTH=$[$CHUNKSIZE/$BLOCKSIZE*($NUMDEVS-$NUMPARITY)]

	echo setting stride to $STRIDE blocks \($CHUNKSIZE KB\)

	echo setting stripe-width to $STRWIDTH blocks \($[$STRWIDTH*$BLOCKSIZE] KB\)

	echo tune2fs -E stride=$STRIDE,stripe-width=$STRWIDTH /dev/$MDDEV

fi



# exit

echo $(date): Success >> "$logfile"

exit 0

Cyborg RAT 7 in Arch Linux

2011-03-31T22:25:00.005+08:00

Earlier, I had written a guide to getting the Logitech MX Revolution working in Linux. Some time ago the friction-free scroll mode broke down on me, so I decided to switch to a new mouse: the RAT 7 (this happened quite some time ago; I just got lazy in writing this). I won’t argue my purchase decision — you either like it or you don’t — but if you have one and want to get it working, then read on.

Details of how to use xbindkeys and xmacro are detailed in the above-linked guide, and I will not repeat it here.

The cursor doesn’t respond!
From a very informative bug report on xorg-xserver-input-mouse, the problem was tracked down to the mode selection button next to the left-click button. Apparently, it sends a release event for button 13, followed by a click event for button 14, each time you press it. In this manner, it cycles through buttons 13, 14 and 15 with repeated presses. X server jams during the start-up process when the mouse does this, and the only fix I know of now is to disable those 3 buttons. This can be done with xinput (detailed in the bug report, but session-specific), or in /etc/X11/xorg.conf. Edit the InputDevice section to look something like this:

Section "InputDevice"

# generated from default
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "auto"
Option "Device" "/dev/psaux"
Option "Emulate3Buttons" "no"
Option "ZAxisMapping" "4 5"
EndSection

Section "InputClass"
Identifier "Mouse Remap"
MatchProduct "Saitek Cyborg R.A.T.7 Mouse"
MatchDevicePath "/dev/input/event*"
Option "ButtonMapping" "1 2 3 4 5 6 7 8 9 10 11 12 0 0 0 16 17 18 19 20 21"
EndSection

ZAxisMapping sets the scroll directions for the scroll wheel, and is not needed to un-jam the mouse. the important line here is Option "ButtonMapping", which disables buttons 13–15 (in bold). Reboot X server (or your system), and that’s all there is to it.

The MatchProduct string is obtained using xinput --list, and should match exactly.

Custom key mappings
For those who want custom button mappings as well, this is the .xbindkeysrc I use for the RAT 7:

###########################
# xbindkeys configuration # For Cyborg RAT 7 Mouse
###########################
#

#Ctrl-PgDn: Thumbwheel right
#using xte (old version): #"xte 'keydown Control_L' 'key Page_Down' 'keyup Control_L'"
"echo 'KeyStrPress Control_L KeyStrPress Page_Down KeyStrRelease Page_Down KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:10

#Ctrl-PgUp: Thumbwheel left
#using xte (old version): "xte 'keydown Control_L' 'key Page_Up' 'keyup Control_L'"
"echo 'KeyStrPress Control_L KeyStrPress Page_Up KeyStrRelease Page_Up KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:11

#Ctrl-F4: Red "Snipe" button
#using xte (old version): "xte 'keydown Control_L' 'key F4' 'keyup Control_L'"
"echo 'KeyStrPress Control_L KeyStrPress F4 KeyStrRelease F4 KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:12

#Alt-Left: Rear thumb button
"echo 'KeyStrPress Alt_L KeyStrPress Left KeyStrRelease Left KeyStrRelease Alt_L' | xmacroplay :0"
m:0x0 + b:8

#Alt-Right: Front thumb button
"echo 'KeyStrPress Alt_L KeyStrPress Right KeyStrRelease Right KeyStrRelease Alt_L' | xmacroplay :0"
m:0x0 + b:9

#
# End of xbindkeys configuration

Addendum (22/08/11): Here’s an alternate .xbindkeyrc using xdotool in place of xmacro:

###########################
# xbindkeys configuration # For Cyborg RAT 7 Mouse
###########################
#
# Version: 0.1.3
#
#Ctrl-PgDn: Thumbwheel right
"xdotool key ctrl+Next"
m:0x0 + b:10
Control+Mod2 + Next

#Ctrl-PgUp: Thumbwheel left
"xdotool key ctrl+Prior"
m:0x0 + b:11
Control+Mod2 + Prior

#Ctrl-F4: Red "Snipe" button
"xdotool key ctrl+F4"
m:0x0 + b:12
Control+Mod2 + F4

#Alt-Left: Rear thumb button
"xdotool key alt+Left"
m:0x0 + b:8
Alt+Mod2 + Left

#Alt-Right: Front thumb button
"xdotool key alt+Right"
m:0x0 + b:9
Alt+Mod2 + Right

#
# End of xbindkeys configuration

Bootable Arch flash drive with separate data partition

2011-03-31T15:00:00.011+08:00

This deceptively simple thing is much more trouble than it seems. Its difficulty can be distilled down to two main reasons:

Removable device woes Windows only being able to read from the first partition on a device that is identified as a removable drive.
More on the removable media bit from WHDC – FAQ for Driver and Hardware Developer:
The removable media device setting is a flag contained within the SCSI Inquiry Data response to the SCSI Inquiry command. Bit 7 of byte 1 (indexed from 0) is the Removable Media Bit (RMB). An RMB set to zero indicates that the device is not a removable media device. An RMB of one indicates that the device is a removable media device. Drivers obtain this information by using the StorageDeviceProperty request.
BIOS troubles “Error 18: Selected cylinder exceeds maximum supported by BIOS” when attempting to boot from the second partition with GRUB bootloader.

This second reason is curious indeed, since my 4GB flash drive (on which this was attempted) has a cylinder count within the 1024-cylinder limit commonly prescribed:

# fdisk -l /dev/sdc

Disk /dev/sdc: 4004 MB, 4004511744 bytes
124 heads, 62 sectors/track, 1017 cylinders, total 7821312 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000bc9f3

Device Boot Start End Blocks Id System
/dev/sdc1 2048 3911679 1954816 b W95 FAT32
/dev/sdc2 * 3911680 7821311 1954816 83 Linux

Can anyone shed some light on this?

In any case, moving on, we find that we need a bootloader that can circumvent this. Here, GRUB exits stage left, and extlinux enters stage right. With much reference to brain0’s comprehensive Arch Linux USB Install and Rescue Media article, I managed to get Arch install media put into the second partition (/dev/sdc2). I quote the relevant section for my future reference here:

The bootable flag needs to be set on sdb2, the boot partition. Now create filesystems on the partitions:
# mkfs.vfat -n SOMELABEL /dev/sdb1
# mkfs.ext2 /dev/sdb2
# tune2fs -i0 -c0 -m0 /dev/sdb2
# e2label /dev/sdb2 ARCH_201005
Notice the label of the Linux partition: While it is arbitrary, it must match the label in the bootloader configuration, which is ARCH_201005 by default. We are finished with the FAT part now: The way we set it up, even Windows will be able to recognize and use it properly. Now, mount the archiso image and the Linux partition:
# mkdir -p /mnt/{archiso,usbboot}
# mount -o loop,ro archlinux-2010.05-netinstall-dual.iso /mnt/archiso
# mount /dev/sdb2 /mnt/usbboot
Copy all the contents of the archiso image onto the USB and umount the ISO:
# cp -a /mnt/archiso/* /mnt/usbboot/
# umount /mnt/archiso
All that is left to do is set up a bootloader. We will use extlinux, as we will be able to reuse all existing configuration from isolinux. First, install the syslinux package, if you don’t have it already:
# pacman -S syslinux
Now, remove the old isolinux bootloader, rename the configuration file, install the extlinux bootloader and umount:
# rm /mnt/usbboot/boot/isolinux/isolinux.bin
# mv /mnt/usbboot/boot/isolinux/isolinux.cfg /mnt/usbboot/boot/isolinux/extlinux.conf
# extlinux --install /mnt/usbboot/boot/isolinux/
# umount /mnt/usbboot
For simplicity, I didn’t rename the isolinux folder here, although the bootloader is technically not isolinux anymore. Feel free to rename it to extlinux if you feel the urge. One last step is setting up a MBR that will recognize the active flag of the second partition and boot its boot sector:
# cat /usr/lib/syslinux/mbr.bin > /dev/sdb

We can do much more than this though. The install media (archiso.img) boots without a root partition on the host, but we can also install a fully functional Arch system alongside the install media, and choose between the two at boot via extlinux. I couldn’t get the booted install media to install to its own partition, so you will need to install from another partition (or try alternative method without rebooting).

Important points to remember during this install:

Do not reformat the partition into which you are installing Arch or you will wipe the existing install media.
Do not set up the GRUB bootloader, since we went to all that effort to use extlinux.

We’re not done yet. The Arch install iso image was compiled with USB boot support, but the new kernel (which we just installed) was not. We will chroot into the new setup (mounted on /bootmnt or /mnt; check!) from the install media setup, and recompile the initramfs with support for USB boot.

# mount -t proc proc /bootmnt/proc
# mount -t sysfs sys /bootmnt/sys
# mount -o bind /dev /bootmnt/dev
# chroot /bootmnt

# nano /etc/mkinitcpio.conf

Edit /etc/mkinitcpio.conf to include USB boot support. Ensure that ‘usb’ is placed before ‘autodetect’ in the hooks list. The HOOKS line should look like:

HOOKS="base udev usb autodetect pata scsi sata filesystems"

Now we remake the initramfs image:

# mkinitcpio -p kernel26
==> Building image "default"
==> Running command: /sbin/mkinitcpio -k 2.6.37-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26.img
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [usb]
:: Parsing hook [autodetect]
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26.img'...SUCCESS
==> SUCCESS
==> Building image "fallback"
==> Running command: /sbin/mkinitcpio -k 2.6.37-ARCH -c /etc/mkinitcpio.conf -g /boot/kernel26-fallback.img -S autodetect
:: Begin build
:: Parsing hook [base]
:: Parsing hook [udev]
:: Parsing hook [usb]
:: Parsing hook [pata]
:: Parsing hook [scsi]
:: Parsing hook [sata]
:: Parsing hook [filesystems]
:: Generating module dependencies
:: Generating image '/boot/kernel26-fallback.img'...SUCCESS
==> SUCCESS

Still not done yet. Let’s make some fstab changes.

#nano /etc/fstab

On a flash drive install, we want to minimise writes to the system. Hence, we add the noatime mount option to the root mount, and also put /tmp on tmpfs. It would also be nice to make the data partition (the first partition, the FAT32 one) accessible in our Arch boot, so let’s do it.

# cat /etc/fstab
#
# /etc/fstab: static file system information
#
#file system      dir         type    options        dump pass
devpts            /dev/pts    devpts  defaults       0    0
shm               /dev/shm    tmpfs   nodev,nosuid   0    0
tmpfs             /tmp        tmpfs   defaults       0    0
UUID=964f5668-ddfe-4b6e-b3c8-aa1f1a854310 / ext2 defaults,noatime  0      1
LABEL=CRUSTOR     /crustor    vfat    user,defaults  0    0

Here, I mounted the data partition by label, but you can do it by UUID too.
Once fstab changes are made, on to extlinux.conf changes.

#nano /boot/extlinux/extlinux.conf

We want to add boot options for regular Arch, and optionally the fallback image as well. These look like below in extlinux.conf:

LABEL archboot
TEXT HELP
Boot into the Arch Linux installation.
ENDTEXT
MENU LABEL Arch Linux
LINUX /boot/vmlinuz26
APPEND root=/dev/disk/by-uuid/964f5668-ddfe-4b6e-b3c8-aa1f1a854310 ro
INITRD /boot/kernel26.img

Note the /dev/disk/by-uuid/ line. This ensures that the flash drive will be able to boot properly in various PCs and setups. /dev/sda* format is not used here since it varies depending on the number of disks on the host system.

The final result should look like this (comments and less important lines stripped):

# cat /boot/extlinux/extlinux.conf
DEFAULT archboot
PROMPT 0
TIMEOUT 50
UI vesamenu.c32

MENU WIDTH 78
MENU ROWS 8
MENU TITLE Arch Linux
MENU BACKGROUND /boot/splash.png

LABEL archboot
MENU LABEL Arch Linux
LINUX /boot/vmlinuz26
APPEND root=/dev/disk/by-uuid/964f5668-ddfe-4b6e-b3c8-aa1f1a854310 ro
INITRD /boot/kernel26.img

LABEL archfallback
MENU LABEL Arch Linux Fallback
LINUX /boot/vmlinuz26
APPEND root=/dev/disk/by-uuid/964f5668-ddfe-4b6e-b3c8-aa1f1a854310 ro
INITRD /boot/kernel26-fallback.img

LABEL archiso
MENU LABEL Arch Linux LiveUSB
KERNEL /boot/vmlinuz26_archiso
APPEND initrd=/boot/archiso.img archisolabel=archboot tmpfs_size=75% locale=en_US.UTF-8
IPAPPEND 0

LABEL existing
MENU LABEL Boot existing OS
KERNEL chain.c32
APPEND hd0 0

LABEL reboot
MENU LABEL Reboot
KERNEL reboot.c32

LABEL off
MENU LABEL Power Off
COMBOOT poweroff.com

ONTIMEOUT archboot

Yes, I did change the partition label, which is why archisolabel is different.

And we’re done. Exit from chroot:

# exit
# umount /bootmnt/proc
# umount /bootmnt/sys
# umount /bootmnt/dev

Try booting into it now and hope everything goes well.

Pity

2010-10-08T23:32:00.000+08:00

“This is pity, he thought, and then he lifted his head in wonder. He thought that there must be something terribly wrong with a world in which this monstrous feeling is called a virtue.”
—Ayn Rand, The Fountainhead

Google Sketchup in Arch Linux x86_64

2010-07-12T18:38:00.005+08:00

What’s needed
From winetricks:
vcrun6
vcrun2005
dotnet20

From AUR:
lib32-nvidia-utils-96xx (conflicts with lib32-libgl)
do not remove optional dependencies after removing lib32-libgl!

registry key edits (regedit):
[HKEY_CURRENT_USER\Software\Google\SketchUp7\GLConfig\Display]

"HW_OK"=dword:00000001

Then run the installation files.
Most of the requirements needed in wine seem to be based around the Visual C++ runtime components, and 32-bit OpenGL compatibility.

SmartyASS: SmartyPants for Advanced Substation Alpha scripts

2010-06-28T19:35:00.011+08:00

Inspired by SmartyPants (from John Gruber), I decided to modify its code in order to fix typography in ASS-formatted scripts. This script does pretty much the same thing as SmartyPants, but leaves non-dialogue text and ASS tags alone, working its magic only on dialogue text.

Naturally, it also faces the same limitations as SmartyPants: apostrophes at the start of sentences cannot be properly replaced. Users are advised to verify the output after running this script. But as a bonus, it removes italic tags.

To use, just run the command “SmartyASS.pl <path-to-ASS-file>” in the directory containing SmartyASS.pl (or in any directory, if the directory containing SmartyASS.pl is in the $PATH environment variable). It can be used as a shell script, possibly requiring some editing of the first line to point at the correct path of the perl binary.

SmartyASS requires perl to run. Which version of it I do not know; I am only a one-day perl coder.

SmartyASS.pl

A brief summary of branches of Physics

2010-05-05T21:27:00.009+08:00

The Nature of Physical Reality, Henry Margenau

This table should be distributed to all Physics first-year undergraduates; Thereupon they are made to memorise it and reproduce it on demand.
The experienced physicist will find this a trivial exercise.

Boosting Western Digital Advanced Format drive performance in Linux

2010-02-27T10:33:00.003+08:00

Once again, a little hdparm trickery seems to be the trick to getting the WD Caviar Green “EARS” series to perform as expected.

From the hdparm man page:

-m

Get/set sector count for multiple sector I/O on the drive. A setting of 0 disables this feature. Multiple sector mode (aka IDE Block Mode), is a feature of most modern IDE hard drives, permitting the transfer of multiple sectors per I/O interrupt, rather than the usual one sector per interrupt. When this feature is enabled, it typically reduces operating system overhead for disk I/O by 30-50%. On many systems, it also provides increased data throughput of anywhere from 5% to 50%. Some drives, however (most notably the WD Caviar series), seem to run slower with multiple mode enabled. Your mileage may vary. Most drives support the minimum settings of 2, 4, 8, or 16 (sectors). Larger settings may also be possible, depending on the drive. A setting of 16 or 32 seems optimal on many systems. Western Digital recommends lower settings of 4 to 8 on many of their drives, due tiny (32kB) drive buffers and non-optimized buffering algorithms. The -i flag can be used to find the maximum setting supported by an installed drive (look for MaxMultSect in the output). Some drives claim to support multiple mode, but lose data at some settings. Under rare circumstances, such failures can result in massive filesystem corruption.

The WD10EARS has a MaxMultSect value of 16.

$ hdparm -i /dev/sdc

/dev/sdc:

 Model=WDC WD10EARS-00Y5B1, FwRev=80.00A80, SerialNo=WD-WMAV51402824
 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
 BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=1953525168
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: Unspecified:  ATA/ATAPI-1,2,3,4,5,6,7

 * signifies the current active mode

$ hdparm -m /dev/sdc

/dev/sdc:
multcount     =  0 (off)
$ hdparm -m 16 /dev/sdc

/dev/sdc:
setting multcount to 16
Use of -m is VERY DANGEROUS.
Only the old IDE drivers work correctly with -m with kernels up to at least 2.6.29.
libata drives may fail and get hung if you set this flag.
Please supply the --yes-i-know-what-i-am-doing flag if you really want this.
Program aborted.
$ hdparm --yes-i-know-what-i-am-doing -m 16 /dev/sdc

/dev/sdc:
setting multcount to 16
multcount     = 16 (on)

Before “-m 16”: 40-50MB/s writes
After “-m 16”: 90MB/s writes

Now that's more like it.

Linux and Western Digital Advanced Format Drives

2010-02-27T10:15:00.002+08:00

As some hard drive buyers might be aware, 4K-sector hard drives have already appeared in the market. Anandtech has a quick primer on these Advanced Format drives, and how they're different.

One point to nitpick on, though: while Linux is capable of creating aligned partitions, not all its partitioning tools do so by default. GParted creates an msdos partition table on new drives by default, and it starts at LBA 63. The result of this is a noticeable slowdown in file transfer to and from the Advanced Format drive.

At the time of this post, the only drives available with 4K-sectors are the Western Digital Caviar Green Series, designated with the model name “EARS” (the older models are labelled “EADS”). If you are using one of these new drives under Linux, you should personally ensure that the partition starts on a mod-8 LBA in order to avoid unpleasant performance loss. A quick guide is available over at the WDC forums.

A summarised snippet of the code:

parted /dev/DEVICE_NAME
(parted) mklabel gpt
(parted) unit s
(parted) mkpart primary ext4 40 -1
(parted) quit

mkfs.ext4 -T largefile4 /dev/DEVICE_NAME

I have written a short snippet on boosting performance of Advanced Format drives in Linux by aligning partitions in the next post.

Logitech MX Revolution in Arch Linux

2010-02-06T23:27:00.005+08:00

There is a multitude of guides out there describing various ways of getting the Logitech MX Revolution mouse working in various flavours of Linux. I figured one more guide wouldn't hurt.

Here, I describe one method using xbindkeys and xmacro.

xbindkeys

From the man page: XbindKeys is a program that grab keys and mouse button events in X and starts associated shell command.

In other words, xbindkeys lets you map keystrokes and mouse buttons to shell commands. What is less obvious is the fact that there are programs that can emulate keystrokes and mouse buttons as well; we will cover this later. What we want to do now is to create a file that defines this mapping.

Begin by installing xbindkeys. In Arch Linux, this is done with the following command:

pacman -S xbindkeys

Next, we create a sample xbindkeys configuration file in your home directory, and open it.

xbindkeys --defaults > ~/.xbindkeysrc

This is what it looks like:

# For the benefit of emacs users: -*- shell-script -*-
###########################
# xbindkeys configuration #
###########################
#
# Version: 1.8.3
#
# If you edit this file, do not forget to uncomment any lines
# that you change.
# The pound(#) symbol may be used anywhere for comments.
#
# To specify a key, you can use 'xbindkeys --key' or
# 'xbindkeys --multikey' and put one of the two lines in this file.
#
# The format of a command line is:
# "command to start"
# associated key
#
#
# A list of keys is in /usr/include/X11/keysym.h and in
# /usr/include/X11/keysymdef.h
# The XK_ is not needed.
#
# List of modifier:
# Release, Control, Shift, Mod1 (Alt), Mod2 (NumLock),
# Mod3 (CapsLock), Mod4, Mod5 (Scroll).
#

# The release modifier is not a standard X modifier, but you can
# use it if you want to catch release events instead of press events

# By defaults, xbindkeys does not pay attention with the modifiers
# NumLock, CapsLock and ScrollLock.
# Uncomment the lines above if you want to pay attention to them.

#keystate_numlock = enable
#keystate_capslock = enable
#keystate_scrolllock= enable

# Examples of commands:

"xbindkeys_show"
control+shift + q

# set directly keycode (here control + f with my keyboard)
"xterm"
c:41 + m:0x4

# specify a mouse button
"xterm"
control + b:2

#"xterm -geom 50x20+20+20"
# Shift+Mod2+alt + s
#
## set directly keycode (here control+alt+mod2 + f with my keyboard)
#"xterm"
# alt + c:0x29 + m:4 + mod2
#
## Control+Shift+a release event starts rxvt
#"rxvt"
# release+control+shift + a
#
## Control + mouse button 2 release event starts rxvt
#"rxvt"
# Control + b:2 + Release

##################################
# End of xbindkeys configuration #
##################################

It is clear that the key bindings follow the following format:

"[shell command to run]"
[keystroke to capture]
[keysym (alternate key descriptor)]

An example:

#Middle-click
"echo 'ButtonPress 2 ButtonRelease 2' | xmacroplay :0"
m:0x0 + c:225
XF86Search

In the above example, key 225 (the search key, which can be found on some multimedia keyboards with more than the standard 108 keys) is also known by the key symbol 0x1008ff1b, or XF86Search. "m" stands for modifier, and is used it you want to add a modifier key (Ctrl, Alt, Shift, and so on) to the keystroke.

You can safely do without the key symbol in the above key binding definitions as well, for example:

#Ctrl-Alt-PgUp
"echo 'KeyStrPress Control_L KeyStrPress Alt_L KeyStrPress Page_Down KeyStrRelease Page_Down KeyStrRelease Alt_L KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:6

Here, "b" stands for button, and we'll be using this to capture mouse button input. For defined keys, use the "c" descriptor instead, as in the earlier example. For moer key binding definitions, use the command xmodmap -pk (you will need to install xmodmap first.)

In contrast with the key bindings, the mouse bindings are much simpler. A list of mouse button bindings for the Logitech MX Revolution is shown below.

1: Left click
2: Scrollwheel click (middle-click)
3: Right click
4: Scrollwheel up
5: Scrollwheel down
6: Scrollwheel tilt left
7: Scrollwheel tilt right
8: Thumb button down
9: Thumb button up
13: Thumbwheel down
15: Thumbwheel up
17: Thumbwheel click

To find out how mouse buttons for other mice are mapped, install and run xev. On execution, xev creates a window with a black-bordered box. Move your mouse cursor inside the box and press one of the buttons. In the console output, look for a section like

ButtonPress event, serial 34, synthetic NO, window 0x3a00001,
root 0x142, subw 0x0, time 12161652, (111,59), root:(1851,1005),
state 0x0, button 5, same_screen YES

ButtonRelease event, serial 34, synthetic NO, window 0x3a00001,
root 0x142, subw 0x0, time 12161652, (111,59), root:(1851,1005),
state 0x1000, button 5, same_screen YES

That tells you which number the button is binded to.

With the above information, it is easy to create key bindings for xbindkeys to capture. Next, we figure out how to run shell commands that will emulate key presses. We will do this with xmacro.

xmacro

To simulate key presses or mouse clicks with xmacro, we use commands in the following format:


 echo 'KeyStrPress  KeyStrPress  KeyStrRelease  KeyStrRelease ' | xmacroplay :0

The section between the single-quotes (' ') describes the command which is sent to xmacro to be run.

I used the following key mappings in my .xbindkeysrc:

#Middle-click
"echo 'ButtonPress 2 ButtonRelease 2' | xmacroplay :0"
m:0x0 + c:225
XF86Search

#Ctrl-Alt-PgUp
"echo 'KeyStrPress Control_L KeyStrPress Alt_L KeyStrPress Page_Down KeyStrRelease Page_Down KeyStrRelease Alt_L KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:6

#Ctrl-Alt-PgDn
"echo 'KeyStrPress Control_L KeyStrPress Alt_L KeyStrPress Page_Up KeyStrRelease Page_Up KeyStrRelease Alt_L KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:7

#Ctrl-PgDn
"echo 'KeyStrPress Control_L KeyStrPress Page_Down KeyStrRelease Page_Down KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:13

#Ctrl-PgUp
"echo 'KeyStrPress Control_L KeyStrPress Page_Up KeyStrRelease Page_Up KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:15

#Ctrl-F4
"echo 'KeyStrPress Control_L KeyStrPress F4 KeyStrRelease F4 KeyStrRelease Control_L' | xmacroplay :0"
m:0x0 + b:17

Modify the above code to suit your own purposes, and insert it into .xbindkeysrc in your home folder.

Now run xbindkeys and check that the key bindings work. Add xbindkeys to your list of autostart applications, and enjoy using your properly mapped Logitech MX Revolution.

Project Osmium: Google Sketchup

2009-10-18T13:20:00.013+08:00

Google Sketchup’s default shading model is far from fantastic, but works fine for drafting. Here, the effects of the black acrylic side panels are poorly rendered, but the effect should be like dark glass — semi-transparent, with a bit of a mirror finish.
Aluminium is used for the top, back and bottom; acrylic is used for the front and sides.

On the left, two renderings that are hopefully more pleasing to the eye: hidden-frame and X-ray renderings. Hidden-frame rendering looks the nicest in my opinion, although it does obscure some details, and shows no information about texturing or colours. The X-ray renderings are a little cluttered, but the internal components should be easily identifiable.

A quick orientation: The radiator is on top, the PSU is mounted at the bottom, flush with the back.

The motherboard is mounted on a second clear acrylic base (attached to the internal frame), and the CPU socket sits right underneath the PSU (a tight fit for anything, really). Yeah, like an SG03 layout, but more cramped.

Lastly, fan grilles and other orifices have not been modelled yet; Thus far, only the back I/O [plate and PCI slots have been cut. Tubing for watercooling is likewise incomplete, as are the CPU and GPU waterblocks.

Here, you can see the internal frame to which all components are attached. Some sections are still incomplete; I am pondering how to add a support bar to the rear assembly, which holds the PSU and peripheral cards. Still, it should be easily clear how the components are attached, even if the screws have not been modelled and rendered.

These are the components, displayed without the skin and supporting frame. The video card is not shown. (There were no pre-rendered models available, and I didn't not have one on-hand for reference.)

The pump was placed in front due to the lack of mountable surfaces; the SSD is in front so I can possibly illuminate it for some cheesecake night shots.

The radiator assembly and the bottom intake fan actually serve to support the aluminium skins (which will be directly attached to them) as well.

The skin and frame are shown without the components. The skin is made up of 2 pieces of aluminium and acrylic each; aluminium for the top and bottom (which curve around to the back and fold in interesting ways to form it), and acrylic for the base side panel on the motherboard side, as well as the curved front+side panel.

It might be interesting to note that the base panel is attached to the internal frame only via perspex blocks (rectangular blocks, of which there are currently 4; I will need to add a few more), so as to reduce the drilling requirement and make the mirror finish more perfect.

The aluminium panel skins will be attached to the frame via flat countersunk screws, polished to match the anodised aluminium.

Details of the hinge mechanism, as well as other missing details, will follow in future updates.

Project Osmium: Materials and Hardware

2009-10-18T13:14:00.005+08:00

Materials
The materials of choice are black transparent acrylic, and black anodised aluminium (brushed finish). While I am not particularly inclined towards lighting in my computer cases, I felt it would be nice to have some diffuse white lighting in the case, accentuating particular elements. Black acrylic would show this off nicely, while hiding the internal components behind a mirror finish when the case lights are switched off.

Hardware
Small form factor here means a micro-ATX form-factor (socket-1156 mini-ITX boards will be a long time in coming...) In addition, I wanted to minimise cost as far as possible, and since I will not be doing heavy overclocking on this setup, it should prove relatively easy to stay on a small budget. This rig will be mainly used for multimedia playback and heavy multitasking, essentially a do-it-all build (excluding gaming; I am not a PC gamer).

The following key components have been picked out so far.

Intel Core i7-860
Gigabyte P55M-UD2
Corsair XMS3-DHX DDR3
XFX Radeon HD5750

The watercooling elements are tentative and may be replaced by other picks sometime.

Swiftech MCP350 pump
Swiftech MCR220-Rev dual-120mm radiator (built-in reservoir)
Dual Noctua NF-P12 for radiator cooling
Enermax Enlobal Marathon / Magma for air intake & memory DIMM cooling

Project Osmium: Introduction

2009-10-18T12:33:00.006+08:00

–noun Chemistry.
atomic number 76. Osmium is a hard, brittle, blue-gray or blue-black transition metal in the platinum family, and is the densest natural element.

Many months ago, when Lynnfield was announced, my interest in a power-efficient, small-footprint quad-core system was piqued. No northbridge, and individual core control; A quad-core system that stays within the 200W power envelope and yet sits in a case smaller than a mini-tower was unthinkable, then.

My earliest ideas revolved around fitting a Core i7-860 in a Lian-Li MUSE PC-C37 case, cooled by a sealed closed-loop cooler such as the Corsair H50. The optical drive + hard drive cage would be removed and replaced with a segregated cooling tunnel for the 120mm radiator, and a 70mm fan intake would cool the memory DIMMs. This idea bounced around in my head for quite a while, until I started looking up case mods on bit-tech and other sites.

The idea of a scratch-build was very appealing; low-form-factor cases have not gained mass-market appeal yet, so pickings are still slim. The only case I could find that would take a full-sized ATX PSU was the C37, and it was still a little too wide for my taste. I also wanted a case that would stand upright to reduce the desktop footprint, and the C37 didn’t look like it could do that comfortably.

Hence, enter project Osmium. The name was (uncreatively) inspired by the element of the same name; this build aims to maximise performance in as small a volume as possible (highest performance density), and it would be decked out in black, like the rest of the other electronics on my table.

Porting search engines from Firefox to Chrome/Iron

2009-04-03T16:26:00.004+08:00

This is a quick method for those seeking to improve the functionality of Chrome/Iron/other Chromium variants on their system.

If you already have Firefox installed and have been playing around with it, then you probably have quite a list of search engines already added. you can, with some expenditure of effort, bring them over to Chromium/Iron. Here's how.

Browse to your Firefox profile folder. On most systems, this should be something like C:/Program Files/Mozilla/Firefox/profile (I'm using a portable build so I can't verify this).
You should see a folder labelled searchplugins. If you see a number of files with extensions ending in .xml, then you've hit jackpot :) Those files are search engine configurations, and should be named according to the search engine they are configured for.
Right-click the appropriate .xml file, then “Edit” (or just open it in your text editor of choice). Look for a line with the following:
From the same line, copy the URL that comes after template=“...”, without the quotation marks.
Right-click on the Location bar in Chrome/Iron, and select “Edit search engines...”. Click “Add”, and a dialog box for adding a ne search engine should appear.
Type in a name and keyword of your choice in the first two text entry fields. In the third entry field, paste the URL you copied earlier. Replace &$8220;{searchTerms}” in the URL with “%s” (excluding the quotation marks).
Click Ok. You should now be able to search using the search engine in Chrome/Iron, simply by typing in the search keyword, then the search terms to use, separated by a space.

Of Sky Crawlers and Change

2009-03-20T15:58:00.008+08:00

I finally watched The Sky Crawlers (from a source I will not name), and it left me with mixed feelings. It has been a long time since I have thought about so many things after watching anime, so I will pen some of them down.

First, a warning to readers: No, I did not have an outline of this written on a sheet of paper, nor did I plan an introduction, body (replete with elaboration) and conclusion, so I am afraid you will have to suffer some of my brain diarrhoea. Any academic body would tell you this is bad writing style, so do not emulate this. But for a blog... whatever.

Without revealing too much of the story and plot, The Sky Crawlers is a story about Kildren: adolescents doomed (or perhaps blessed) to live the prime of their youth over and over again, in an unchanging landscape. Like with other Mamoru Oshii works, this one is a thinker. What makes this one different is that it's boring, absolutely boring, but intentionally so. Here I should immediately clarify that I do not use the word “boring” with the intent of insult.

Truthfully, the film piqued my interest only after I read Justin's review of it (scathing look at the anime industry? Ooooh...) I don’t fully agree with him. As one of the posters on the ANN forums says, the theme of this movie is so general that it could apply to almost anything. It could be a scathing look at anime... or at engineering, or business, photography, or any number of fields and disciplines that have fallen into the rut of wash-rinse-repeat. From a general perspective, Sky Crawlers is the embodiment of pretty-but-boring; lovely textures and lighting, sharp CG, but flat textures on flat characters, and bland voice-acting. I wonder if this is Oshii’s way of making his point that invariability is an undesired result we should not be aiming for.

As a teenager I sometimes thought to myself, “How nice things would be if they could stay the same so I would never have to grow up”. Now the irony of that statement comes back to bite me. It's triflingly amusing because at that moment in time, I was looking forward to an eternity of constancy; the preservation of a state that includes my preference for an eternally unchanging state of constancy. If that state could have been perfectly preserved maybe I would have been in frozen, time-preserved bliss. Wouldn't that be a dandy state of things?

But in retrospect, perhaps invariability is desirable only in the context of an inevitably changing background. In a world where things are changing slowly but surely, constancy is the flip side of the coin, the greener side of the field. I As a teen, my thoughts aligned with dreams of never growing up or graduating, because growing up and graduating is the de-facto state of things I could not avoid. I wonder what I would have said in an alternate reality where ideas such as graduation never existed.

In the first chapter of The Nature of Physical Reality, (part of) a paragraph reads "Professor William Lyon Phelps, in his charming informal lectures to the undergraduates at Yale, insisted that physics had far less to say about truth and reality than did poetry. and to prove his point he asked them: 'Would you now read a physics text that is 100 years old? Of course not. But you still read Shakespeare!'"

So much for truth and reality then, as convenient and useful constructs of the mind. Maybe they are not constant either, changing as our perceptions and collective ideas do. Perhaps, as the cliché goes, the only thing that doesn't change is change itself. And if change is the only thing we can count on, then it's probably time for me to grow up and stop getting too comfortable in my little academic pigeon-hole.

Fockkkk

2008-06-07T01:39:00.007+08:00

More than 6 years (nay, closer to 7) with Windows XP, and I only just realised that every time I disconnect a USB device, my tablet says “fockkkk”. In a computerish accent, nonetheless quite clearly and distinguishably, “fockkkk”.

Killing It Softly

2008-05-31T02:14:00.004+08:00

I have a habit of trying to hold the power button for the shortest time possible before the computer performs a hard power down. I never gave too much thought to this until today, when I realised that I still feel queasy about holding down that power button. Pressing down on the windpipe of a whirring, breathing animal, until it ceases all activity, while you watch pensively for further stirring before confirming its state of death.