Directory and sub directory sizes and disk usage

Although this belongs to the Linux Command Cheat Sheet, this post is here to list all the Directory / Sub Directory and Disk drive usage

To begin with, if it is disk space you are looking for information about, the following command should display the useful disk usage information

df -h

If you want to know the size of all sub directories in a directory

Move to the directory containing the sub directories (if root, can take a long time)

du -sh *

Now, if you want to know all the directories above a certain size, in this case that measure in GBs, here is how you do it

du -h <dir> | grep '[0-9\.]\+G'

If the above lists directories not within the gigbytes domain, you can use the following instead

du -h <dir> | grep '^\s*[0-9\.]\+G'

Splitting INNODB files to separate .ibd files

Please note: most of you are probably visiting this post for the answer to whether you can use multiple disks with innoDB, the answer is, even though you can not move idb files around like when you move MYI MYD files, you can move an IDB file while the DB engine is not running, then create a symlink where the Innodb expected to find the original file, then start mysql again, so the short answer is that innodb does work with symlinks just like myisam.

The second answer is, you can not use more than one disk on the same table (even though you can for the same database by putting every table on a separate disk).

As you would probably know, innodb, by default stores all data into its central, single file data dictionary (on debian, /var/lib/mysql/ibdata1), you can ask innoDB to put every table’s data in a separate file (tablename.ibd) by simply adding the directive innodb_file_per_table anywhere under [mysqld] section.

Databases already in single file will stay there, the directive only affects the creation of new databases, InnoDB does not mind mixed mode regarding ibd files, dome tables in a single database can be in the shared file, and some can have their own files. files created while the directive is in effect will get separate files, you can then remove the directive from your my.cnf and the engine will start putting new databases back into the shared file.

Mind you, unlike ISAM tables, you can not move these files around, in MyISAM you can simply copy the 3 files that are any table, the .frm, .MYI and .MYD and you have your table elsewhere, in INNODB, this is not the case as much is still stored in INNODB’s ibdata1 , also, innodb files are connected to a certain database with internal strings, Yet, you can copy the whole data directory and have it work, remember, log files are also needed and must be kept.

To move an .ibd file and the associated table from one database to another, use a RENAME TABLE statement:

RENAME TABLE db1.tbl_name TO db2.tbl_name;

The main disadvantage of splitting data files per table is disk space, files in INNODB never get smaller, when in 1 file, other tables can use that space of deleted rows in another table, when split apart, you can only use the occupied space again if you add rows to the same table that had deleted rows.

The advantages are

1- You can utilize multiple disks for both space and performance (By using symlinks)
2- You can backup certain tables and leave others to add flexibility to your backup plan

Extracting Audio from youtube files without loss of quality

To not get you confused with terminology, let’s create our own with the simple words we know.

For those who have been here before and want the lowdown to remind them of the commands, just scroll to the bottom of this post.

this bit of theory i am starting with is not needed for you to extract your audio, but simply to introduce you to what we are doing

The video file has 2 sub files, one for video and one for audio, and they are synchronized together so that the people’s lips movement in the video appear to be speaking what the audio file is playing.

If we do not want to lose any of the audio’s quality by decoding then re-encoding again, we will want to extract the “Inner audio file” without modifying it, and to put it into a separate file (container).

Separating the audio from the video is easy with a free tool called ffmpeg, and here are the exact instructions

1- Make sure ffmpeg is installed on your computer and is added to the system path, if not see the article (installing ffmpeg)
2- Download the youtube or any other video file to c:downloadvideo, assuming we have 2 tutorial files one is tutorial.flv and the other is tutorial.mp4, both were downloaded from youtube.com (if you don’t know how to download a youtube video, see this article).
3- open the command line (Command prompt can be opened from your start menu, look in accessories).
4- Enter the following command into your command prompt to change active directory to where you have your files, in our example we enter the following into the command prompt

cd c:downloadvideo

5-Then, for my first file, i will execute the command

ffmpeg -i tutorial.flv

You should now see, on your command prompt window, what sub files (streams) are inside your container file (the file you downloaded from youtube or anywhere else).

--------------------------------------------------------------
  Duration: 00:02:06.59, start: 0.000000, bitrate: 64 kb/s
    Stream #0.0: Video: flv, yuv420p, 320x240, 29.97 tbr, 1k tbn, 1k tbc
    Stream #0.1: Audio: mp3, 22050 Hz, mono, s16, 64 kb/s
--------------------------------------------------------------

if you don’t see the word mp3 like this example, don’t panic, just move on to step 7.

In our first example file, the audio stream turned out to be an MP3 stream as you can see below, if that was not the case and we had a different format (As i will explain next), we would have had different choices in extracting, but for this example, we have MP3, meaning we do not need to re-encode anything, just copy the stream from the container we downloaded into a new container that will be created

6- Extract the MP3 file without re-encoding and while keeping full quality like in the video

ffmpeg.exe -i tutorial.flv -acodec copy tutorial.mp3

But for some reason, copeying the OGG ogg vorbis from a webm file without reencoding did not work untill i added the -vn switch

ffmpeg -i Bir_G_zellik_Yap_Murat_Dalk_l.webm -vn -acodec copy test1.ogg

So, now i have the file tutorial.mp3 that simply has the same clarity as my video file. the -acodec copy parameter told ffmpeg to just copy into new file, and not to re-encode

7- If it does not say mp3 anywhere in your results, you have a different audio format. that we will deal with now.

so let us deal with our second file tutorial.mp4 that turned out not to have MP3 in it, but rather AAC.

So executing the command

ffmpeg.exe -i tutorial.mp4

Returned the result

--------------------------------------------------------------
Duration: 00:05:02.44, start: 0.000000, bitrate: 281 kb/s
  Stream #0.0(und): Audio: aac, 44100 Hz, mono, s16
  Stream #0.1(und): Video: h264, yuv420p, 320x240 [PAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 50 tbc
--------------------------------------------------------------

In this case, we do NOT have the option that will allow us to get an MP3 without re-encoding, we can re-encode it into MP3 as i will show you in a bit, or we can extract and use an AAC audio file.

An AAC file is not at all bad, sometimes it is better than an MP3. Why ? The AAC file is a newer format and it will still play on many devices, on most computers, relatively new IPODs, Most modern mobile phones and many other MP3 players.

AAC file streams are mostly put into m4a containers, but can also be .m4b, .m4p, .m4v, .m4r, .3gp, .mp4, .aac. (OF YOUR CHOICE, the most common is m4a, but some older mobiles use 3GP)

So, let us first try to extract the AAC stream without re-encoding (next step we will re-encode to MP3 for those who want an MP3)

ffmpeg.exe -i tutorial.mp4 -acodec copy tutorial.m4a

Now, the device you want to use does not support AAC files, so what we can do is ask ffmpeg to convert it to MP3 for us

ffmpeg -i tutorial.mp4 tutorial.mp

—————————————————–

The Lowdown

1- Find out what streams exist

ffmpeg -i tutorial.mp4

2- Copy MP3 stream without re-encoding

ffmpeg.exe -i tutorial.mp4 -acodec copy tutorial.mp3

3- Copy AAC stream without re-encoding

ffmpeg.exe -i tutorial.mp4 -acodec copy tutorial.m4a

4- Re-encode audio to MP3 file fromat

ffmpeg -i tutorial.mp4 tutorial.mp3

 

To encode any file into a DVD compatible file (Best when used with DVD players that have a USB input or with USB TV)

ffmpeg -i 9.mp4 -threads 2 -filter:v "scale='if(gt(a,720/480),720,-1)':'if(gt(a,720/480),-1,480)',pad=w=720:h=480:x=(ow-iw)/2:y=(oh-ih)/2" -target pal-dvd 9.mpg

Sometimes, you might want to replace pal-dvd with ntsc-dvd

Speed testing an internet connection

Well, there are a few ways to check the upload / Download speed of an internet connection, one way is speedtest.net which uses flash to download a file, and upload a file, both to a server close to you

On systems where we do not have a browser or do not have a browser that supports flash, one can download a file (With wget  on Linux for example), the quest would be this

You will need a file that is hosted on a network that you know for fact is faster than your own internet connection, for me, i have been using this one very successfully

cachefly.net 100mb.test

So, on a LINUX system, entering

wget http://cachefly.cachefly.net/100mb.test

On a casual 2.4Mb (That’s Mega Bit not Byte) , it should result in something like this

---------------------------------------------------------------------------------------
--2012-04-19 11:41:09--  http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 140.99.93.175
Connecting to cachefly.cachefly.net|140.99.93.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `100mb.test'

 6% [=>                                     ] 6,897,290    284K/s  eta 5m 41s
---------------------------------------------------------------------------------------

While on a much faster connection i have somewhere else (theoretical 100Mb), the results are like this

--2012-04-19 08:44:20--  http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 140.99.93.175
Connecting to cachefly.cachefly.net|140.99.93.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `100mb.test'

100%[======================================>] 104,857,600 41.2M/s   in 2.4s

2012-04-19 08:44:22 (41.2 MB/s) - `100mb.test' saved [104857600/104857600]

There are also other factors in internet connection speed that i will get to soon, for example, latency, and efficient routing.

things that i will get to when i have the time.

Review LG A190 – Very cheap, yet tough and somewhat convinient

This is probably the cheapest dual sim mobile phone on the market. with one relatively significant drawback.

To give you the lowdown first, this phone has 1 issue concerning how to chose the line before sending or calling, by default you call from line 1, unlike the Dual sim Samsung(s) that i have used, this phone does not provide a simple easy convenient way to switch lines (In some samsung phones there is a dedicated line change button), to switch a to the second line, you must enter the number, then go to options, then hit call via, then chose the line you will use.

The phone does receive to both lines at all times (Dual stand by), so in situations where the second line is only to receive, and very seldom send or call from, this phone will do the trick, considering that this is the cheapest dual sim phone on the market.

The phones hardware seems tough. (and i have been using it heavily for a year and it is very tough)

the phone comes with radio that unlike other brands, has an internal antenna and does not requiere an earpiece to work.

Now that i have given you what i think is most important above, let me get to the other details of this phone.

What is in the box

Phone, Battery, Charger, Users manual, and warranty card (No Earpiece).

Even though the phone charger is a simple Micro-B-USB plug, connecting the phone to a computer will only result in charging the phone from the computer USB power, no data exchange can happen.

The phone comes with only one game called space ball that happens to be very similar to the popular snake game (Like the old one).

The phone can store 500 contacts, and 100 messages, 10 memos and 20 calendar entries

A strange feature in this phone is called fake call or self call, where you set the phone to make a fake call at a certain time, it rings, then you pick p and pretend to talk to someone, in other phones like samsung phones, you can have the phone play a side of a pre recorded phone call, but not this phone, this phone simply rungs, you can chose the name and number of the person ringing you I never really understood the need for this feature, but when they are adding it to the cheapest phones, i should be investigating this fake call feature soon, and what people use it for

Rescueing data of a failed hard drive

I accedientally pulled the power plug of a PC from the socket, and that PC was just starting to boot. The seagate hard drive inside stopped working, and the bad sectors turned out to affect the partition table, in any case, i slaved it on a windows vista PC, then into the Computer management, disk management panel, and what do you know, as if it has no paritions…

The solution to detect the boundaries of the 4 partitions it had is software called XXXX

Ran the software (The analyze option) , and what do you know, my partions exactly, 100MB made by Windows 7, a 479 GB partition for Windows, a 1GB swap partition for linux and an EXT3 partition for Linux…

So happily i asked the software to write the partitioning info to the disk, but the disk won’t hold the data, the bad sectors are where Windows writes the partition information

So, i ran down to the computer shop (In our building), and got the same exact drive (Seagate 500GB Model number xxx)

Mounted both on a Linux machine as slaves, both the damaged and the target.

To find out which one is SDC and which one is SDB, i watched as the linux machine booted, and as it booted, it threw in errors saying SDB all the time, so i know that SDB is the busted drive !

Installed gddrescue (apt-get install gddrescue), and ran it with the following command

ddrescue /dev/sdb /dev/sdc resumelog.log

(The additional log file helps us resume in case of interruption)

Once that is done, i put the new hard drive in a Windows machine, still can not see any partition info

1- Ran xxxx, it can see the 4 partitions, write changes… and what do you know, the partitions stick, we are good to go, i restart, but still, Windows can now see the partitions, but thinks drive G is not formated !

So i opened the command prompt (Elevated), then ran the command

chkdsk g: /f

the /f stands for fix, the thing took some time, but after the restart drive G works fine, all files are in there, and no one wants to kill me no more 🙂

I like my rebranded tp-link router – XTECH Router

NOTE: This post was updated on feb, 1, 2013

Today, i got a new Wireless Lite N router for JD20 (Around $28), and it works like a dream, in reality, the following is a sampleof the routers web GUI response headers

Server: TP-LINK Router
Connection: Keep-Alive
WWW-Authenticate: Basic realm="XTECH 150Mbps Wireless Lite-N Router WR771LN"
Content-Length: 26344
Content-Type: image/jpeg

200 OK

Some Hanania Investment Group seems to be responsible for this new XTECH Brand, and for this router, it is well worth the money.

The RJ45 eithernet cable mentioned on the outside as included is not included, but then again, why would i need 1 more eithernet cable.

This model seems to be the same as TP-WR741ND TP-WR840N (since the TP-WR741ND is the same but with a detachable antenna), and if you know what little endian, big endian, and a hex editor are, you may be able to put the “Open Sourced” TP-LINK software back on this hardware.

The reason you can not is that those routers check for file name when you upload, and only 3 people in the world know what the firmware file name requiered by this router to accept new firmware is. I am about to become the fourth to know and hopefully everyone else can come later, (FYI i dont know who the three people are, i just assumed there must be three people)

Anyway, i hope all other XTECH stuff are as good as this one

NOTE: The only reason you may want to have the TP-LINK firmware back on this router is that there is no mentioning of this router on xtechinternational.com and therefore i can not find a firmware upgrade, TP link will surely have the upgrades as soon as they are out though.

—————–
Naming the firmware update file:

Update, although this is a re-branded TP-Link, i switched my TP-Link to dd-wrt, i think i will switch this to open-wrt or dd-wrt and not to the original tp-link firmware, mind you, we still need to know what the existing firmware expects for a firmware name.

In a TP-LINK, the answer is simple, you download the file factory-to-ddwrt.bin from dd-wrt, then you rename the file to be identical to the update file from tp-link.com, namely wr740nv1_en_3_12_4_up(100910).bin , but for xtech (The re-brand), there is no update file, and therefore we don’t know the name of the acceptable update file.

So lets see the similarities, and see what options we have
tp-link TL-WR740N gets wr740nv1_en_3_12_4_up(100910).bin
X-Tech xt-wr771LN should get ?
wr740nv1_en_3_12_4_up(100910).bin (same: NO)
wr771nv1_en_3_12_4_up(100910).bin (NO)
wr771lnv1_en_3_12_4_up(100910).bin (NO)
xtwr740nv1_en_3_12_4_up(100910).bin (NO)
xt_wr740nv1_en_3_12_4_up(100910).bin (NO)
xt-wr740nv1_en_3_12_4_up(100910).bin (NO)
xt740nv1_en_3_12_4_up(100910).bin (NO)
xt_wr771lnv1_en_3_12_4_up(100910).bin
xtwr771lnv1_en_3_12_4_up(100910).bin
wr771lnv1_en_3_12_4_up(100910).bin
—————–

The router is probably an atheros chip router with a

HYNIX 933S C
HY5DU561622FTP-D43 RAM

If anyone wants some photos, i can post them here for you no problem

you can find this router almost everywhere, SmartBuy Carefour, and others are stacking piles of it.

Good Luck

S.M.A.R.T (SMART) data of hard drives under linux

The results of the commands on this post are Here

On windows, You can use speedfan, under linux you can read the smart data as follows

Use the smartctl command, but first you must install the package

apt-get install smartmontools

1- Checking for smart support, (All recent hard drives have it but you need to enable it in BIOS)

smartctl -i /dev/sdb

2- Enable reading it

smartctl -s on -d ata /dev/sdb

See it

smartctl -d ata -H /dev/sdb

Read more

smartctl -d ata -a /dev/sdb

Now some manufacturer specific smart atributes exist, you need to google them out, Also smart is not definitive, if S.M.A.R.T data says ok but there is a ticking noise in your hard drive, Don’t trust the hardware data, get backup.

Some hard drives like the Samsung Spinpoint come with extensive diagnostics software, My 3 2.5′ disks pass the test, pass the smart test, and tick when warm, i will let you know if it was the tick of death sometime soon