Disk load and IO wait on linux

Using the top command, you can find out how much time your processor spends waiting for input and output from disk, the IO wait field, but how can we know what applications are responsible for that IO wait.

to examin disk io and iowait more closely, we need some tools, the following is a list of tools that can show you exactly what is causing all the IO wait so that you can start from there.

Sometimes, apps that use the disk with many random reads should go to the secondary disk. but to find that application, take a look at the following tools, i will be explaining each and every one of them as we go.

Here are a bunch of programs on linux that will tell you just that

  • iotop – apt-get install iotop – simple top-like I/O monitor
  • iostat – apt-get install sysstat – shows and monitors activity per disk – The sysstat package contains the sar, mpstat and iostat commands for Linux.
  • isag – apt-get install isag – Interactive System Activity Grapher for sysstat
  • dstat – apt-get install dstat – versatile resource statistics tool
  • vmstat 1
  • iostat 1
  • lsof
  • strace -e trace=open
  • strace -e trace=open -p
  • In ps auxf you’ll also see which processes are are in uninterpretable disk sleep (D) because they are waiting for I/O.

examples

iostat -c -d -x -t 2 100

Gigabit Ethernet over USB 3 with a hub

I got myself a new C850B797 toshiba laptop with an i7-3630QM processor and 16GB of ram (coz it’s cheap), the C850B797 is the grey edition in the C850 series, the processor is a 22nm Lithography and everything is very good as far as i can see, but then i realized that the laptop had one very serious issue, i am limited to 100mbit Ethernet, and i really do move plenty of files around all the time (many gigabytes), so i needed to get gigabit, but how ?

Usb speed is 480mbit/second, much of that goes overhead, so the Ethernet controller would probably be USB3 which would block my only usb port, so the answer is to get a USB3 hub and connect the USB gigabit ethernet to it, but it turns out that the hub followed by gigabit eithernet is already on the market from one of 2 providers.

The first, the one i got is the StarTech ST3300U3S USB hub and Ethernet controller, that uses ASIX AX88179 for gigabit Ethernet after a VIA VL811 USB3 hub, the question is, will i be getting the speeds i am hoping for from this ?

I will be running tests concerning reliability of the combo (since both are connected in serial) and let you guys know.

1- StarTech ST3300U3S USB hub and Ethernet controller – Power adapter included = $55, Also happens to be the same exact thing as the SYBA inf@zone (infozone) with the same chips and the same plastic casing, the SYBA infozone will sell at $33 if you buy 3 together, i found 3 for $98.99 on ebay, making it much cheaper than the $66 each for the same exact thing from StarTech, SYBA labels it as SY-HUB50056.)

AX88179 — USB3.0 to 10/100/1000M Gigabit Ethernet Controller
VIA VL811 – SuperSpeed USB 3.0 Hub Controller

The other options on the market are…
2- Kanex DualRole Gigabit Ethernet with 3-Port USB Hub (USB3GBITX) – Power adapter not included – $40
AX88179 — USB3.0 to 10/100/1000M Gigabit Ethernet Controller
No idea about the hub chip yet.

Please bare in mind that this same combo but with USB 2 and a 100Mbit eithernet adapter should not cost more than $10, it is the USB 3 and gigabit Ethernet that are expected to be this expensive.

Via also produces the VL812 which is almost identical to VL811

Disk spindown in linux, specifeying spindown idle time

Disk Spin down

There is a probelm with disk spindown via hdparm, the problem is that you must address a disk as /dev/sdc , which changes in the case of USB media and other disks, even when you add slaves,

hdparm -Y /dev/sdb will spin a disk down instantly

hdparm -S 240 /dev/sdb will set this disk to sleep when idle for 20 minutes (5 second units here)

or adding at the bottom of the file /etc/hdparm.conf a section such as

/dev/sdc {
spindown_time = 240
}

to make those changes persistent across reboots.

To check the status of a disk, here is what you do

hdparm -C /dev/sde

You could get one of the following results
When spun down…
drive state is: standby
When active
drive state is: active/idle

Don’t make your disks spindown too often, 20 minutes is good for me almost in all circumstances.

If the disks don’t spin down, chances are that selftest is enabled…

Check if it is enabled with

smartctl -a /dev/sdb
if it reads
Auto Offline Data Collection: Enabled.

then you need to disable it with
smartctl –offlineauto=off /dev/sdb

then wait for them to finish (if a test is running) then spin down.

ZoneMinder on debian with a DVR card using techwell chipset

1- Installed the debian base system, debian wheezy (7) (But should work with debian squeeze (6))
2- apt-get install ssh openssh-server fail2ban
3- INSTALLED THE HARDWARE 8 CAMERA DVR CARD into the PCI slot
4- Modified the network interfaces to use a static IP, and pointed the DMZ of the router to that IP
5- installed some software (Even though some of them would have been installed anyway when i install zoneminder), but i do it anyway just in case they are optional in zoneminder

apt-get update
apt-get install apache2 php5-mysql libapache2-mod-php5 mysql-server
apt-get install ffmpeg libarchive-tar-perl libarchive-zip-perl libdate-manip-perl libdevice-serialport-perl
apt-get install libjpeg62 libmime-perl libstdc++6 libwww-perl zlib1g
apt-get install zoneminder

I can see close to the end of operation

ZoneMinder is stopped
invoke-rc.d: initscript zoneminder, action “status” failed.
Starting ZoneMinder: success

I hope it’s nothing too serious

Now, check if your card is supported, you issue lspci -v as root… Since my 8 port card is supported, i see eight stanzas of this

I got eight of the following labeled 03:02.0 03:02.1 03:02.2 03:02.3 03:02.4 03:02.5 03:02.6 03:02.7

03:02.2 Multimedia video controller: Techwell Inc. TW6816 multimedia video controller (rev 10)
Subsystem: Techwell Inc. TW6816 multimedia video controller
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 3
Memory at febfe800 (32-bit, non-prefetchable) [size=1K]
Capabilities: [44] Power Management version 2

If you dont see your card’s devices, well, your out of luck, maybe you should try a newer kernel, i am using 3.2 from debian wheezy (7)

Now, to the second test

ls -ls /dev/video*

NO love, nothing, If i were lucky, i would get something like

ls -ls /dev/video*
0 crw-rw----+ 1 root video 81,  0 30. Jul 15:18 /dev/video0
0 crw-rw----+ 1 root video 81,  2 30. Jul 15:18 /dev/video1
0 crw-rw----+ 1 root video 81,  4 30. Jul 15:43 /dev/video2
0 crw-rw----+ 1 root video 81,  6 30. Jul 15:43 /dev/video3
0 crw-rw----+ 1 root video 81,  8 30. Jul 15:17 /dev/video4
0 crw-rw----+ 1 root video 81, 10 30. Jul 15:17 /dev/video5
0 crw-rw----+ 1 root video 81, 12 30. Jul 15:17 /dev/video6
0 crw-rw----+ 1 root video 81, 14 30. Jul 15:17 /dev/video7

But i did not, so i need the driver to my techwell card.

Simple, there is a nice person who made a driver already, let’s see how we can install it

First, we need GIT so that we can download the driver

apt-get install git
apt-get install linux-headers-$(uname -r)
apt-get install make sudo

(without installing sudo, the compile gives the error /bin/sh: 1: sudo: not found which is because the guy probably wrote the driver with ubuntu in mind)

git clone git://gitorious.org/tw68/tw68-v2.git
cd tw68-v2
make
make install

So, i was expecting 8 cameras, i got the following

ls -ls /dev/video*
0 crw-rw---T 1 root video 81, 0 May 12 23:12 /dev/video0
0 crw-rw---T 1 root video 81, 2 May 12 23:12 /dev/video1
0 crw-rw---T 1 root video 81, 4 May 12 23:12 /dev/video2
0 crw-rw---T 1 root video 81, 6 May 12 23:12 /dev/video3

probably a driver issue, oh well, lets see what we can do with 4 then i will see about this issue once i have 4 working:D

ln -s /etc/zm/apache.conf /etc/apache2/conf.d/zoneminder.conf
adduser www-data video
/etc/init.d/apache2 restart

Now, visiting 192.168.2.101/zm/ allows me to add cameras, the matter is very intuitive from that point on, all you need is a browser, remember you will need to password protect things too :D so that people from the outside can not look at your cameras.

Wheezy is out, so is openVZ, but LXC seems to be in !

Yes, Wheezy is out to the public, and openVZ is out of Wheezy, so what to do.

Basically, what i am doing now is investigating the alternative LXC, i have no time to learn right now, so i am going to have to do this fast.

I have a gut feeling that LXC is better than openVZ, after all, it is in the mainline kernel, and it is supposed to be marvelously easy to install, so let me start working on this with everyone here.

NOTES: if you want to give away LXC containers to people, you will need to use AppArmor with it, here, i run my containers, so i will not be installing AppArmor in this tutorial, but maybe soon i will add a tutorial for the AppArmor part.

So, LXC here we come, to completely replace openVZ, with something more open (Sorry Parallels Virtuozzo, welcome IBM), something that can keep up with the kernel and not keep us behind.

I will be turning this post into a tutorial on installing and running LXC on debian wheezy (7) with memory allocation to containers and with the kernel that shipped with wheezy, i should be done creating this tutorial in a few days, and it will remain an incremental effort where i will be adding more and more as i learn about this.

NOTES: memory allocation is not compiled with the kernel by default but disabled, you enable it by adding a parameter to grub. (Not anymore, now memory allocation works out of the box)

1- Install base system of wheezy (debian 7)

2- Install some stuff i can never do without

apt-get update

apt-get upgrade

apt-get install ssh openssh-server fail2ban

fail2ban is a very important application that will prevent outsiders from bruit force cracking your server, it is very important, without it you will be hacked sooner or later (especially if you are in a datacenter), hackers look for servers to send spam from all the time.

Now, we need to specify a hostname for this machine (the LXC HOST), i want to call mine server5.example.com

echo server5.example.com > /etc/hostname

/etc/init.d/hostname.sh start

hostname

hostname -f

apt-get install ntp ntpdate

Now, we need to setup networking for LXC, every physical NIC (Network adapter) will need a bridge.

To create a bridge, you need to install

apt-get install bridge-utils

Then your /etc/network/interfaces file must look like this

------------------------------------------------
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
  auto lo
  iface lo inet loopback
# The primary network interface
  #allow-hotplug eth0
  #iface eth0 inet dhcp
#Bridge setup
auto br0
  iface br0 inet static
  bridge_ports eth0
  bridge_fd 0
  address 192.168.2.121
  netmask 255.255.255.0
  gateway 192.168.2.1
  dns-nameservers 8.8.8.8
------------------------------------------------

apt-get install lxc

You will be presented with the following prompt, i myself accept the default /var/lib/lxc

Please specify the directory that will be used to store the Linux Containers. If unsure, use /var/lib/lxc (default). LXC directory:

mkdir /cgroup

Add the following line in /etc/fstab using a text editor:

cgroup /cgroup cgroup defaults 0 0

mount -a

Now, to make sure everything is working like it should

lxc-checkconfig

------------------- OUTPUT OF lxc-checkconfig ----------------START

Kernel config /proc/config.gz not found, looking in other places...
Found kernel config file /boot/config-3.2.0-4-amd64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroup: enabled
Cgroup clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled
Macvlan: enabled
Vlan: enabled
File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig.

------------------- OUTPUT OF lxc-checkconfig ------------------END

And on the host machine, you need to enable IP forwarding befor you fire up any of those LXC containers

 echo 1 > /proc/sys/net/ipv4/ip_forward

But to make that peppermint you need to edit the file /etc/sysctl.conf where we can add a line containing net.ipv4.ip_forward = 1

/etc/sysctl.conf:

net.ipv4.ip_forward = 1

You might find that the entry is already there but with the value 0, in that case just flip the zero to a 1, or you might find it there but commented out, in that case, delete the # that precedes that line to enable it.

To enable the changes made in sysctl.conf (And you don’t if you already executed the echo 1 statement above) you will need to run the command:

sysctl -p /etc/sysctl.conf

Now that LXC is officially installed, there is more than 1 way to create containers, debootstrap is one of them (you will need to install it, and the container config will need to be done manually by adding a few lines into a file you create inside the container area), while i will use the LXC way by using the application lxc-create you are free to use any tool, including importing containers from vmware (copying vmware containers will work).

Also worth mentioning, i use apt-cacher so when i am asked about the urls of the distro, i simply modify it to read http://192.168.2.133:3142/ftp.us.debian.org/debian/ which is how i accerss apt-cacher to speed up things and not re-download everything every time.

So, lets start

lxc-create -t debian -n vm33

On a newer releast (7.7), the above gave me an error, so the following was the error and the solution (needed command)


MIRROR=http://ftp.us.debian.org/debian lxc-create -n vm10 -t debian -- -r wheezy

Or if you want to use apt-cacher

MIRROR=http://192.168.10.237:3142/ftp.us.debian.org/debian lxc-create -n vm10 -t debian -- -r wheezy

1- Preseed file anyone? Enter (optional) preseed file to use: <== leave this one empty

2- Chose the distro (debian wheezy for me)

3- 64 or 32, i use 64

4-
Archives.

[*] Debian Security

[*] Debian Updates

[*] Debian Backports

[ ] Debian Proposed Updates

5- Mirror.

i modify this to read http://192.168.2.133:3142/ftp.us.debian.org/debian/ in order to use my apt-cacher, you can put any mirror here, or leave the default one (http://ftp.debian.org/debian/ Mirror Security http://security.debian.org/ and Mirror Backports) provided for you. Archive areas Main, Packages (leave blank or specify the packages you want, you can install them later with apt-get), then the root password

You must keep in mind that even after you see the message ‘debian’ template installed ‘vm33’ created, the config file for vm33 is not really ready, you need to enable networking in it manually. so, let’s edit the file /var/lib/lxc/vm33/config and add networking support

vi /var/lib/lxc/vm33/config

NOTE: THE BELOW IS FOR TYPICAL SETUPS, FOR HETZNER DATACENTER, PLEASE SEE THE POST ON LXC NETWORK SETUP WITH HETZNER.

then add the lines right before #Capabilities and after the lines of ## Container

lxc.network.type = veth

lxc.network.flags = up

lxc.network.link = br0

lxc.network.name = eth0

lxc.network.ipv4 = 192.168.2.125/24

Also, before we start the container, there are a few things we need to do…

there seems to be an issue with the ssh keys, so what we will do around this issue is copy the keys from the host, (We will generate new ones for the conatiner later)

EXECUTE ON HOST

cp /etc/ssh/ssh_host_dsa_key /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key
cp /etc/ssh/ssh_host_dsa_key.pub /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key.pub
cp /etc/ssh/ssh_host_ecdsa_key /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key
cp /etc/ssh/ssh_host_ecdsa_key.pub /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key.pub
cp /etc/ssh/ssh_host_rsa_key /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key
cp /etc/ssh/ssh_host_rsa_key.pub /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key.pub

Then, they won’t work without proper permissions

chmod 0600 /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key.pub
chmod 0600 /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key 
chmod 0600  /var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key

Now i reboot the server just to be on the safe side, then i do the following

lxc-start -n vm33 -d
lxc-info -n vm33

When you run the command for information, you should see the word RUNNING and a pid.

Just SSH to the host !

Now if you want to create new host keys for SSH just do the following

delete the files

/var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_dsa_key.pub
/var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_ecdsa_key
/var/lib/lxc/vm33/rootfs/etc/ssh/ssh_host_rsa_key

execute

dpkg-reconfigure openssh-server

—————————————

Making LXC auto start at the system boot
The old Way – create a symbolic link, should still work, but i have not tried

ln -s /var/lib/lxc/vm34/config /etc/lxc/auto/vm34_config

The new way that provides better control of the order they are started in.
Set lxc.start.auto == 1 in the config

Then, the following will tell the system what containers to start first, and when

PHP code protection, obfuscation, and encoding

Update:

BCOMPILER is no longer supported past PHP 5.3, BUT, the same developer went back to developing APC, and now, there are APC functions that can do what bcompiler used to do.
I will come back to explain how to use it later (it works perfectly for me without the original files all together. the trick is 2 functions, apc_bin_load and apc_bin_dump.

The other good option is PHC (of phpcompiler.org).

A third option would be obfusc.com

End of update (2014-05-10)
—————————–

For encoding, there are 3 options

Before i waste your time, what i do is this, i obfuscate the code so that the function names, variable names, and other things are useless and meaningless to the reader, then i use bcompiler. Now here are the choices and programs i used through the years.

Zend Guard (formerly Zend Encoder)- Expensive ($600) – Some say they have cracked it but i have never seen any evidence of that, requires the free Zend Optimizer or Zend Guard Loader to run the encoded code
Ioncube – I have a license for this one, does what zend encoder does, needs it’s loader on the server to work, provides a cheaper encode-online edition for those with small projects, lowest license is for 200 dollars
bcompiler – Free, and probably faster than the other two, it is simply native PHP opcode, meaning you spare PHP the need to compile your code (interpret), making your code run faster, but reversing it is not very hard. to make reversing harder, you need to obfuscate your code (change variable names and function/method names to make the code unreadable, read below for software to obfuscate that are available online.
BENCODER v1.6 – Encode your PHP script using bcompiler – to facilitate using bcompiler

PHP Obfuscators

Here are just a few, if you google PHP obfuscator you will find hundreds 😀

truebug – $45
codeeclipse – free service, obfuscation happens online on the website
phpprotect.info – Simple free tool that obfuscates variable names, which is enough most of the time, it is very hard to read code with non descriptive variable names
iddin.sourceforge.net – all in German, but translating reveals that it is an obfuscator