WordPress, Varnish, nginx: The SSL https detection problem

I have a certain setup with a Varnish5 caching reverse proxy, with nginx to terminate SSL connections.

The problem with this setup is that wordpress can’t detect https, hence, it can not enforce it, nor will it link the CSS accordingly etc…, and if your blog’s address starts with https, you have a problem.

there is more than one solution that i will enumerate here, I chose to force all connections to come through https (The first), but there are others, the one i will present here (Which i did not use) resolves the problem and allows wordpress to detect whether we are on a secure connection or not.

And even though this post-article talks about wordpress, everything here can apply to any PHP application (Or even other applications written in different programming languages)

1- Enforce https for the whole website

The simplest way to solve the problem (Which i chose) is to redirect all traffic to https.

in the varnish script, implement the following

in the sub vcl_recv, implement the following

if (req.http.host ~ "^(www\.)?example\.com$") 
{
	if (req.http.host ~ "^(?i)example.com" || req.http.X-Forwarded-Proto !~ "(?i)https") {
		return (synth(750, ""));
	}
}

Now, the following section is the sub vcl_synth

sub vcl_synth {
    if (resp.status == 750) {
        set resp.status = 301;
        set resp.http.Location = "https://www.example.com" + req.url;
        return(deliver);
    }
}

Once the above is in place in the varnish script, You will need to tell wordpress that it’s all HTTPS (SSL)

We do that in the config file, the only file we know that does not change when we update

define('FORCE_SSL_ADMIN', true);
$_SERVER['HTTPS']='on';

Now, the above should do it if you don’t mind that your website only works in https, if you do want http to remain an option, then here is another solution

Start by creating a phpinfo() page to check whether you have the $_SERVER[‘HTTP_X_FORWARDED_PROTO‘] variable , if you do, your work is partially done, but you need to mind too things

the varnish server needs to only cache the https copy (because most browsers won’t allow mixed content these days, and if the css is linked to as http, it won’t display correctly), you can do that easily with something like the following in the vcl_hash area

if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }

And then, also in the wp-config file, add the following (probably almost anywhere)

if ( isset( $_SERVER['HTTP_X_FORWARDED_PROTO'] ) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https')
        $_SERVER['HTTPS']='on';

Enable adoptable storage for the J5 J500H (3G)

Allowing Adoptable storage on the Samsung J5 20165 – J500H

There are a few tricks for this particular phone
1- We need to replace the recovery system with TWRP

Enable phone developer mode, then USB debugging and OEM override (Disable protection)
Download ODEN
Download the TWRP .TAR file for your device, for the Samsung J5 2016 (J500H), you should find i here
https://twrp.me/samsung/j53gxx.html

reboot your system into bootloader mode (Vol down, Home and power at the same time for 3 seconds to start the phone)

Open ODEN, chose the tar file for AP, and upload the TWRP

Now, don’t let the phone reboot into the system or it will overwrite the bootloader with the original one, instead
once the phone starts rebooting, hit, Volume UP and home button and power at the same time, and TWRP will patch the
system once it opens in recovery mode so that it won’t overwrite the new TWRP

Reboot system, and boot with Power, Home, And volume down button for bootloader change

From this point on, you can enter into TWRP with the Home, Vol UP and Power button combination at boot time

Now download the patch to enable adoptable storage and store it to the SD card (adoptable_storage_enabler_ashyx) from here https://forum.xda-developers.com/galaxy-tab-a/how-to/patch-adoptable-storage-enabler-t3507387

Start the recovery mode (TWRP mode)

Hit install and chose the file you have downloaded

That’s all you need

Now, next thing you will need to do is to format the SD card using the guide here

Storing whatsapp media (video, photo, etc) to SD card.

Backup or move an LXC container

You can not just copy an LXC container, wat you need to do is among the lines of

cd /var/lib/lxc
tar --numeric-owner -cf container_125_fs.tar vm125

The magic is mainly about maintaining the numeric owner of files, which is lost when you copy the files using the host machine ūüėČ

you can untar it the same way you untar any other tar file.

Storing whatsapp media (video, photo, etc) to SD card.

To have android treat an SD card like internal storage, you will need to format it as internal storage, a feature of android disabled by most manufacturers, so here is how to do it even when the button is missing.

After you get a RELIABLE big SD card, let us get the procedure out of the way, Read below about reliability and dangers etc.. before you decide on getting a cheaper SD card

This feature of android is called Adoptable Storage.

1- On your windows PC (Or mac): Download adb.exe, there are many places to download it on the internet, but for convenience, here is a 64 bit download of the adb.exe file, in my case, it came with Android Stdio, and i found it in C:\Users\yazeed\AppData\Local\Android\Sdk\platform-tools, obviously you will have your username in place of yazeed in the path above

2- On your phone: Open your phone, Go to Settings, About, Software information, And tap repeatedly on Build Number until it informs you that you are now a developer

3- On your phone: A new developer options Menu should appear, Open it and enable USB debugging

4- Connect the phone to the computer via USB cable

5- On your windows PC (Or mac): Open the command prompt (CMD/Shell), I ran it as administrator to avoid any surprises, and run the following commands, obviously, you will need to change the directory to wherever you put the adb.exe file

 

cd C:\Users\yazeed\AppData\Local\Android\Sdk\platform-tools
adb shell
sm list-disks
** disk:179,64 <= as you can see, this handle/identifier is our SD card inside the phone
sm partition disk:179,64 private

6- Disconnect the wire.

7- Go to settings => storage => sd card (where yo normally format an SD card), then from the three button menu on the top right, select Migrate data, this will move the existing data from internal to SD card.

So right now, We have formatted the SD card as internal storage, the way this works is that the SD card is overlayed over the existing storage, so whatever you already have on your internal memory is not masked by mounting a new SD card in it’s place, but whatever you save from this point on will be saved to the SD card.

Pros Cons
* The SD card size is added to the internal phone storage so you can install apps on the SD card and have apps store data on SD card when they don’t offer the option to do that natively

* The internal phone flash memory is not consumed as fast due to flash wear, even though phones usually fail for other reasons before the flash failiure occures, it is mentioned here for completeness

* You can not plug this SD card into your computer, in fact, you are expected to never ever remove it from your phone, this is to avoid catastrophic failiure and data loss

* If your SD card fails or otherwise goes bad, data loss will not only affect the data, but also afects your applications and settings, making the damage much worse

* A slow SD card might affect your phone’s performance.

 

There are a few things to keep in mind, I ordered them by importance, most important first.

  • ADB.EXE is a file from google, part of the development tools, and not a third party tool, it’s for android debugging by the people who make android
  • *RELIABLE: The SD card you chose must be super reliable as failure of the card will result in loss of your data, applications, settings, and your phone’s functionality in general, making it much worse than when using the SD card as external storage, Get an SD card with wear leveling, as the ext2/ext4 file system that android uses is not too good with flash memory, I personally use sandisk, some sandisk models explicitly state that they have wear leveling, get those, and beware, there are many many fake SD cards on the market, so get it from a reliable source.
  • *FAST: The SD card you chose needs to be fast, the faster the better, even though at some point, the speed of the SD card hits the phone’s limit (the speed the phone can access the SD card), having a very fast SD card will ensure fastest possible operation of your favorite apps
  • *One partition: I have seen people online split the SD card into some internal storage, and some external, but they provide no justification to the benefits of this, as windows will only read the first partition of the disk, My advice is to use the whole SD card as internal, as ther benefit of having part of it as external storage is debatable at best.In case you are wondering how that is done, in place of the line where it says
    sm partition disk:179,64 private

    You will need to use

    sm partition disk:179,64 mixed 20

    The alternative line you see above will create an internal storage partition of 80% and leave 20% as external SD storage

  • *WHATSAPP: The main reason i did this to my phone is that my whatsapp messenger is bombarded with gigabytes of video all the time, and whatsapp does not natively allow for storing media files such as video to external SD cards, in this solution, there is no compromise as the solution is invisible to whatsapp, whatsapp now stores media to the SD card with no real compromise.
  • Why is it disabled in most phones: This is a good question, The official reason, which is a valid, legitimate reason, why companies like samsung asus and others disable this feature is reliability, manufacturers can’t guarantee that yo did your homework and got a reliable and fast SD card, they also can not guarantee you will not accidentally switch the phone on after removing the card, or even remove the card while the phone is running, so in short, leaving this feature in is unwise for the average user, a great side effect (To them) from this legitimate reason is that they can sell larger capacity phones at a higher price and a higher profit margin, but that is irrelevant as the first reason is very legitimate.
  • *Learn: To learn more about the technology, you can visit the Adoptable tech page on the official android website, https://source.android.com/devices/storage/adoptable
  • * Temporarily taking it out: Most phones these days come with Dual SIM, and in most cases, an SD card slot that takes the place of the second SIM card, the issue here is that if you ever need to use the second SIM card slot for a SIM card, you can not just temporarily pull the SD card out without causing massive damage to your apps and data, god knows how the system and it’s sb systems will react to missing data and applications, probably by recreating them blank again, and when you insert the SD card again, there will be a clash of file names.

Searching for text within file contents

Searching for text within file contents in Linux

When you want to do this, odds are, you know what kind of file you are looking for, My first guess would be you would not be looking in zip, mp4, flv, etc… those are huge files, where in the unlikely event they do contain your string, it’s still not the file you are looking for…

So, you would probably want to look at the problem as, search the contents of text files for a certain text string

the best way to acheive that is to start by allowing the find command to find text files that may contain that string, then passing the “candidate for searching” files to the grep command

So if I am looking for a config file that contains a config named living-room, I would use a search such as this one

find /hds -name '*.conf' -exec grep -i 'living-room' {} \; -print

What is nice about this is that you can also look at both separately, so the find command has plenty of documentation online, and so does grep

Now, there are other ways to do this, the most popular of which is using grep on it’s own, here are some examples

The directives are as follows

i stands for ignore case (Slows things down, but sometimes necessary).
R stands for recursive. (Look inside inner folders depth first)
l stands for "show the file name, not the result itself".
e look for things matching a patern
--include Include files that match this pattern
--exclude Exclude files that match this pattern
--exclude-dir exclude directories listed
Examples
grep -Ril "text-to-find-here" /
grep --include=*.{c,h} -rnw '/path/to/somewhere/' -e "pattern"
grep --exclude=.o -rnw '/path/to/somewhere/' -e "pattern" grep --exclude-dir={dir1,dir2,.dst} -rnw '/path/to/somewhere/' -e "pattern"




Hope this helps, good luck

3.3V Power supply – Linear Voltage Regulator

As usual i will start with the main points / the lowdown

1- Car voltage can (Depending on car) can go up to 15.3 when bulk charging the battery, the maximum voltage for the LD1117AV33 is 15V, so it is a no go.

2- My DoIt esp12F devkit came fitted with the AMS1117.

The regulator that shipped with my DoIT esp12F devkit is the AMS1117, which seems to be an excellent choice providing up to 1A of current, and accepting input voltages of up to 18V ! Yet, the voltage regulator is not fitted with a heat sink, hence, it will provide nowhere close to 1A reliably

the three liniar regulators I will be comparing in this post are

 

 ld1117v33  lf33abv  TC1262
Maximum Input Voltage 15V 16V 6V
Minimum input voltage (Based on dropout voltage) 4.3V 4.3V 4.3V
Can we change the input voltage while application is switched on  Yes   Yes   Yes
Maximum current draw (Is it enough for my application)
Higher is better
800 500 500
Quiescent current
Current drawn with no load (Not too relevant to cars)
Lower is better
5.5 mA  0.5 mA  0.070 mA

 

Even though i have always used the LD1117AV33 Linear Voltage Regulator in IOT projects, simply because it can provide up to 800 MA of current at 3.3V, I had to look at other voltage regulators for this particular project because it involves a vehice, Now, you may want to skip to the comparison table if your IOT project is not for a car, but

 

As usual, I post my research in this blog for my own use and so that others can benefit from it, so sometimes it looks as though it is not well written, but it could spare you a long time of research

First thing you need to know, is that not all cars have the same voltage, the¬† 12V car battery voltage is nominal, a car battery is usually full and in good health when it’s voltage is around 12.8 or even 12.9 (while the car is off), and because the charging circuit differs from car to car,¬† A car alternator might step up the voltage to up to 15.3 when it is bulk charging the battery, and float charge a car battery at 14.2 or 14.3 when it considers it full, so bottom line, a linear voltage regulator needs to tolerate at least 15.5V

 

Even though i have always been using the LD1117AV33 step down, and indeed it is a reliable thing, I can not find any document that states that going above the maximum input voltage of 15V is okay, hence, the regulator is disqualified from the word go

 

 

 

Godaddy / Wild west domains, reseller examples

Godaddy / Wild west domains, reseller examples (List is work in progress)

fxdomains.com
polodomains.com
hostingdude.com
cheap-domainnames.com

Super resellers (Sells reseller accounts)
DomainResellerDirect.com

 

The main purpose of this list is for my own reference, as I try to make my own reseller account work for me, as i investigate how they are making it happen, i will probably learn a bit from them.

GPS / u-blox and Adafruit-Mediatek

The things i add here are primarily for my reference, but they could save you a lot of time, so here i am only covering the practical side, wikipedia can help you with the theory. things here are sort of in random order, so feel free to use your browser’s search facility.

I have the following positioning systems/chips/boards, so most of the stuff you will find here will relate to them

* V.KEL VK2828U7G5LF TTL Ublox GPS module with antenna: supports GPS, GALILEO, SBAS (WAAS, EGNOS, MSAS, GAGAN)
* K-172 USB GPS USB Receiver Dongle Adapter Smart Antenna Module For Gmouse Glonass, also running u-blox, Support NMEA 0183 and ublox binary protocol.
* The ADAFRUIT ultimate breakout board with the MTK3339 chip (Not U-BLOX), but rather mediatek

A-GPS and AssistNow:

The long story short: A-GPS seems to be when the satellite data is available in advance, whether you allow your system to download it from the internet as it goes, or you download it in advance and make it available to the system, it boils down to telling your system where the satellites are and their relevant data rather than wait for your device to download that data from the satellites at very low speeds (Sat data is downloaded at a maximum of 50 bits per second, so getting the data over the internet or from the SD card in your raspberry pi or arduino etc is much much faster).

AssistNow is how the U-Blox A-GPS works

1- A-GPS only work out of the box¬†with u-center, the software by u-blox, on your raspberry PI or arduino, it will not work out of the box , to make it work you need to look online for software that does that (It’s on github)

the online edition is when the data is downloaded from the internet in real time (when it is needed), the AssistNow¬† offline is when you download “AlmanacPlus¬ģ Differential Almanac Correction Data” from the internet and store it with your device (On whatever is driving the device, be it a PC, laptop, raspberry pi, arduino etc…)

2- The adafruit GPS raspberry pi board does A-GPS in a different way, which works out of the box, they add a battery to the board, and once data is downloaded once, it stays on the board for as long as there is a battery ūüėČ

 

 

Finding duplicate files on NAS storage

I have this bad habit of copying things so many times when modifying them, when that is a large database, we talking many gigas, so here is a script to find those duplicate files among many hard drives and telling you which ones are duplicates, moving and deleting and symbolic linking is done manually after.

1- this script is PHP-CLI, so make sure that is installed on your computer
2- this script runs the find command, make sure it can execute that program
3- you run the script with the path parameter, but will need to edit the script to change the 1GB size i have hard coded

What this script does is

1- find files with size greater than 1GB (find /hds -size +1G)
2- Store the files in database with size
3- retrieve the files ordered by size
4- if 2 files have exactly the same size, calculates MD5sum for the first MB of the file
5- If the MD5 of the first MB of the files are the same, calculate the whole MD5
6- If they turn out to be duplicates, they are printed to the command line