About Kubernetes and it's Architecture

Kubernetes more frequently pronounced as (K8s) it’s an open source container orchestration platform designed for running distributed applications and services at scale. It was originally designed by Google and is now maintained by the Cloud Native Computing Foundation

Kubernetes components can be logically split up into these two categories:

• Master (Control Panel): These components run on the master nodes of the cluster and form the control plane.
• Node (Worker): Applications and services running on the worker nodes to receive instructions and run containers.

Master (Control Plane) has following components. These components are responsible for maintaining the state of the cluster:

1. etcd
2. API Server.
3. Controller Manager
4. SchedulerEvery worker node consists of the following components. 

Every worker node consists of the following components. These components are responsible for deploying and running the application containers.

1. Kubelet
2. Container Runtime (Docker etc)

These components are responsible for deploying and running the application containers.

Let’s discuss more about Master components one by one

etcd 

It is a simple, key value storage which is used to store the kubernetes cluster data (such as number of pods, their states, namespaces etc). in simple words, it is the database of kubernetes. It is only accessible from the API server for security reasons. etcd enables notifications to the cluster about configuration changes with the help of watchers. Notifications are API requests on each etcd cluster node to trigger the update of information in the node’s storage.

kube-apiserver

Need to interact with your Kubernetes cluster? Talk to the API. The Kubernetes API is the front end of the Kubernetes control plane, handling internal and external requests.The kube-apiserver is responsible for API validation for all resource creation requests before the resources are actually generated and saved to the data store. Users can communicate with the API server via the kubectl command line client or through REST API calls.

kube-controller-manager

Kubernetes manages applications through various controllers that operate on the general model of comparing the current status against a known spec. These controllers are control loops that continuously ensure that the current state of the cluster (the status) matches the desired state (the spec). They watch the current cluster state stored in etcd through the kube-apiserver and create, update, and delete resources as necessary. kube-controller-manager is shipped with many controllers such as:

• Node Lifecycle controller
• DaemonSet controller
• Deployment controller
• Namespace controller

kube-scheduler

Since Kubernetes is an orchestration framework, it has builtin logic for managing the scheduling of pods. The kube-scheduler component is responsible for this. Scheduling decision depends on a number of factors such as:

Resource requirements of the app
Resource availability across nodes
Whether the pod spec has affinity labels requesting scheduling on particular nodes
Whether the nodes have certain taints/tolerations preventing them from being included in the scheduling process

The kube-scheduler takes all of these considerations into account when scheduling new workloads.

cloud-controller-manager

The cloud-controller-manager is responsible for managing the controllers associated with builtin cloud providers. These controllers are dedicated to abstracting resources offered by individual cloud providers and mapping them to Kubernetes objects.
Cloud controllers are the main way that Kubernetes is able to offer a relatively standardized experience across many different cloud providers.

Node Components

The node, or worker, machines are the primary workhorses of Kubernetes clusters. While the master components handle most of the organization and logic, the nodes are responsible for running containers, reporting health information back to the master servers, and managing access to the containers through network proxies.

Container Runtime(Famously Docker)

The container runtime is the component responsible for running the containers within pods. Rather than being a specific technology as with the other components, the container runtime in this instance is a descriptor for any container runtime that is capable of running containers with the appropriate features. Docker, CRI-O and containerd are some of the popular options for container runtimes.

kubelet

The kubelet component is an agent that runs on every worker node of the cluster. It is responsible for managing all containers running in every pod in the cluster. It does this by ensuring that the current state of containers in a pod matches the spec for that pod stored in etcd.

kube-proxy

The kube-proxy component is a network proxy that runs on each node. It is responsible for forwarding requests. The proxy is somewhat flexible and can handle simple or round robin TCP, UDP or SCTP forwarding. Each node interacts with the Kubernetes service through kube-proxy.

Containers vs Virtual Machine vs Docker

Container

Using containers, everything required to make a piece of software run is packaged into isolated containers. Unlike VMs, containers do not bundle a full operating system - only libraries and settings required to make the software work are needed. This makes for efficient, lightweight, self-contained systems and guarantees that software will always run the same, regardless of where it’s deployed. Docker is an example of container technology, there are many other likes containerd, CRI-O etc

Virtual Machine(VM'S)

Virtual machines (VMs) are an abstraction of physical hardware turning one server into many servers. The hypervisor allows multiple VMs to run on a single machine. Each VM includes a full copy of an operating system, one or more apps, necessary binaries and libraries - taking up tens of GBs. VMs can also be slow to boot.

Containers vs Virtual Machines

Containers and virtual machines have similar resource isolation and allocation benefits, but function differently because containers virtualize the operating system instead of hardware, containers are more portable and efficient

Docker

Docker is the world's leading software containerization platform, there are many other  Docker’s website describes it as “an open-source engine that automates the deployment of any application as a lightweight, portable, self-sufficient container that will run virtually anywhere.”.

• Docker is built on top of LXC, and therefore runs containers, not VMs as VirtualBox for instance 
• Docker containers are made of portable “images”, similar to LXC/VZ templates, but much more powerful (versionning, inheritance …) 
• Docker “images” can easily be created via Dockerfile’s, where are set the base image and the steps to run in order to create your image 
• Docker allows to have run multiple instances of your container without needing to copy the image (base system) files 
• Docker daemon (which manages / runs LXC containers) provides a REST API used by Docker CLI utility … but this REST API can be used by any application 
• Docker runs on virtually all operating systems (Linux, Mac OS, Windows …) and platforms (Google Cloud Platform, Amazon EC2) 

Docker is the world's leading software containerization platform.

Changing MySQL Data Disk/Directory on Linux

First stop Mysql service

/etc/init.d/mysqld stop
rename the mysql directory
mv /var/lib/mysql /var/lib/mysql_old

Mount a secondary disk as data disk, I have mounted /disk2 to my /dev/sdb

create a folder ‘mysql’ in destination disk/partition

copy whole Mysql data directory to ‘mysql’ in preserve mode

cp –pr /var/lib/mysql/ /disk2/mysql

-p means preserve the specified attributes (default: mode, ownership, timestamps)


then create a symbolic link from new data directory to old data directory

ln -s /disk2/mysql/ /var/lib/mysql/

Change permissions on /var/lib/mysql.

chown -R mysql:mysql /var/lib/mysql

If AppArmor is not enabled you do not need to do below. But it is recommended to have AppArmor enabled for security. Follow below steps to allow mysql access through AppArmor:

vim /etc/apparmor.d/usr.sbin.mysqld and add the data directory to it
/disk2/database/mysql/** rwk,
 /disk2/database/mysql/ r,

Now start Mysql service
/etc/init.d/mysqld start
If its working properly then delete each file and folder from old data directory

cd /var/lib/mysql_old rm –rf  *

NOTE: Please take a backup of your MySQL databases before attempting migration.

How to Redirect HTTP to HTTPS in Apache

Apache’s mod_rewrite makes it easy to require SSL to be used on your site and to gently redirect users who forget to add the https when typing the URL. Using Apache to redirect http to https will make sure that your site (or a part of it) will only be accessed by your customers using SSL.  This is better than using SSLRequireSSL because users often forget to type in the https and will be automatically redirected.

Before you can set up an Apache redirect from http to https, you will need to do the following:

Make sure your SSL certificate is successfully installed so you can access https://www.yoursite.com (for more information see our Apache SSL Installation instructions)
Make sure mod_rewrite is enabled in Apache
Now you just need to edit your httpd.conf file or the file where your virtual host is specified and add these lines to redirect http to https:

RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}

In many cases, you can also just add those lines to a file named .htaccess in the folder that you want to redirect http to https.

Now, when a visitor types http://www.yoursite.com/mypage.htm the server will automatically redirect http to https so that they go to https://www.yoursite.com/mypage.htm

Note: You can also redirect a single page from http to http in Apache by using this in your configuration file or .htaccess file:

RewriteEngine On
RewriteRule ^apache-redirect-http-to-https\.html$ https://www.yoursite.com/apache-redirect-http-to-https.html [R=301,L]

Run sudo a2enmod rewrite to enable it. Restart apache using sudo service apache2 restart and provided your .htaccess syntax is correct this will work.

https://www.sslshopper.com/apache-redirect-http-to-https.html

Slow query logs in AWS MySQL RDS

The MySQL slow query log and the general log can be written to a file or a database table by setting parameters in your DB parameter group. You must set these parameters before you can view the slow query log or general log in the Amazon RDS console or by using the Amazon RDS API, Amazon RDS CLI, or AWS SDKs.

You can control MySQL logging by using the parameters in this list:

slow_query_log: To create the slow query log, set to 1. The default is 0.
general_log: To create the general log, set to 1. The default is 0.
long_query_time: To prevent fast-running queries from being logged in the slow query log, specify a value for the shortest query execution time to be logged, in seconds. The default is 10 seconds, the minimum is 0. If log_output = FILE, you can specify a floating point value that goes to microsecond resolution. If log_output = TABLE, you must specify an integer value with second resolution. Only queries whose execution time exceeds the long_query_time value are logged. For example, setting long_query_time to 0.1 prevents any query that runs for less than 100 milliseconds from being logged.
log_queries_not_using_indexes: To log all queries that do not use an index to the slow query log, set to 1. The default is 0. Queries that do not use an index are logged even if their execution time is less than the value of the long_query_time parameter.

log_output option: You can specify one of the following options for the log_outputparameter.

TABLE (default)– Write general queries to the mysql.general_log table, and slow queries to the mysql.slow_log table.
FILE– Write both general and slow query logs to the file system. Log files are rotated hourly.
NONE– Disable logging.

When logging is enabled, Amazon RDS rotates table logs or deletes log files at regular intervals. This measure is a precaution to reduce the possibility of a large log file either blocking database use or affecting performance. FILE and TABLE logging approach rotation and deletion as follows:

When FILE logging is enabled, log files are examined every hour and log files older than 24 hours are deleted. If the remaining combined log file size after the deletion exceeds a threshold of 2 percent of a DB instance's allocated space, then the largest log files are deleted until the log file size no longer exceeds the threshold.

When TABLE logging is enabled, log tables are rotated every 24 hours if the space used by the table logs is more than 20 percent of the allocated storage space or the size of all logs combined is greater than 10 GB. If the amount of space used for a DB instance is greater than 90 percent of the DB instance's allocated storage space, then the thresholds for log rotation are reduced. Log tables are then rotated if the space used by the table logs is more than 10 percent of the allocated storage space or the size of all logs combined is greater than 5 GB. You can subscribe to the low_free_storage event to be notified when log tables are rotated to free up space.

When log tables are rotated, the current log table is copied to a backup log table and the entries in the current log table are removed. If the backup log table already exists, then it is deleted before the current log table is copied to the backup. You can query the backup log table if needed. The backup log table for the mysql.general_log table is namedmysql.general_log_backup. The backup log table for the mysql.slow_log table is named mysql.slow_log_backup.

You can rotate the mysql.general_log table by calling the mysql.rds_rotate_general_log procedure. You can rotate the mysql.slow_logtable by calling the mysql.rds_rotate_slow_log procedure.

Table logs are rotated during a database version upgrade.

* from AWS Documents

VirtualBox kernel modules do not match this version error (RTR3InitEx failed with rc=-1912 (rc=-1912) )

I have successfully fixed the issue in my Ubuntu 16.04.1 LTS
Check the package
dpkg -l | grep -i virtualbox

Uninstall from system
sudo apt-get remove unity-scope-virtualbox
sudo apt-get update && sudo apt-get remove --purge virtualbox-5.1

wget -q https://www.virtualbox.org/download/..._vbox_2016.asc -O- | sudo apt-key add -
wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -

Finally install it:

sudo apt-get update
sudo apt-get install virtualbox-5.1


This fixed the issue for me 

Generating Self Signed Certificate

When using the SSL Endpoint feature for non-production applications, you can avoid the costs associated with the SSL certificate by using a self-signed SSL certificate. Though the certificate implements full encryption, visitors to your site will see a browser warning indicating that the certificate should not be trusted.

In order to successfully install an SSL certificate you need the following things

CSR file
Private key
Certificate key

I am assuming the below,

Domain Name: testssl.com

A private key and certificate signing request are required to create an SSL certificate. I am going to use 'openssl req' to generate the certificates.

Initiate a CSR

openssl req -new -newkey rsa:2048 -nodes -keyout testssl.key -out testssl.csr

After this command you need to provide the details about the organization and what type (wildcard or subdomain) of certificate you are going to generate. You can hit enter for a “challenge password”, leaving the password empty.



Now you have completed generating the private key and the CSR.

This will create below files

testssl.csr--will be used in creating certificate
testssl.key--> Private key (need to convert in pem format)

Generate SSL certificate
The self-signed SSL certificate is generated from the testssl.key private key and testssl.csr files.

openssl x509 -req -days 365 -in testssl.csr -signkey testssl.key -out testssl.crt

The testssl.crt file is your site certificate for use with SSL add-on along with the testssl.key private key.

Applying on HAProxy:

In most cases, you can simply combine your SSL certificate (.crt or .cer file provided by a certificate authority) and its respective private key (.key file, generated by you). Assuming your certificate file is called testssl.crt, and your private key file is called testssl.key, here is an example of how to combine the files:

cat testssl.crt testssl.key > testssl.pem
sudo cp testssl.pem /etc/ssl/private/

This creates the combined PEM file, called example.pem and copies it to /etc/ssl/private. As always, be sure to secure any copies of your private key file, including the PEM file (which contains the private key).

If the above method doesn't work then you can convert your file to PEM format like belwo:

openssl rsa -in testssl.key -outform PEM -out testssl.pem

openssl x509 -inform PEM -in testssl.crt -out testssl-cert.pem

Combined these two and put in HAProxy config file e.g.

cat testssl-crt.pem testssl.pem > testssl.pem

Edit HAProxy config file and add the certificate

vim /etc/haproxy/haproxy.cfg

frontend localhost
    bind *:80
    bind *:443 ssl crt /etc/ssl/private/testssl.pem
    mode http
    default_backend nodes

Installing the AWS Command Line Interface on Linux

This section discusses various ways to install the AWS CLI.

Note:
The AWS CLI comes pre-installed on the Amazon Linux AMI. Run sudo yum update after connecting to the instance to get the latest version of the package available via yum. If you need a more recent version of the AWS CLI than what is available in the Amazon updates repository, uninstall the package (sudo yum remove aws-cli) and then install using pip.

Note:
The awscli package may be available in repositories for other package managers such as APT, yum and Homebrew, but it is not guaranteed to be the latest version. To make sure you have the latest version, use one of the installation methods described here.

like installation on ubuntu you can execute the below command
$sudo apt-get install aws-cli

Install the AWS CLI Using Pip

Pip is a Python-based tool that offers convenient ways to install, upgrade, and remove Python packages and their dependencies. Pip is the recommended method of installing the CLI on Mac and Linux.

Prerequisites

Windows, Linux, OS X, or Unix
Python 2 version 2.6.5+ or Python 3 version 3.3+
Pip
First, check to see if you already have Python installed:

$ python --version

Install the AWS CLI Using pip

With Python and pip installed, use pip to install the AWS CLI:

Linux, OS X, or Unix

$ sudo pip install awscli

To upgrade an existing AWS CLI installation, use the --upgrade option:

$ sudo pip install --upgrade awscli
Pip installs the aws executable to /usr/bin/aws. The awscli library (which does the actual work) is installed to the 'site-packages' folder in Python's installation directory.

Install the AWS CLI Using the Bundled Installer (Linux, OS X, or Unix)

If you are on Linux, OS X, or Unix, you can also use the bundled installer to install the AWS CLI. The bundled installer handles all the details in setting up an isolated environment for the AWS CLI and its dependencies. You don't have to be fluent in advanced pip/virtualenv usage, nor do you have to worry about installing pip.

Prerequisites

Linux, OS X, or Unix
Python 2 version 2.6.5+ or Python 3 version 3.3+
Check your Python installation:

$ python --version

Install the AWS CLI Using the Bundled Installer

Follow these steps from the command line to install the AWS CLI using the bundled installer.

To install the AWS CLI using the bundled installer

Download the AWS CLI Bundled Installer using wget or curl.

Unzip the package.

Run the install executable.

On Linux and OS X, here are the three commands that correspond to each step:

$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
$ unzip awscli-bundle.zip
$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
If you don't have unzip, use your Linux distribution's built in package manager to install it, typically with either sudo yum install unzip or sudo apt-get install unzip.

Configuring the AWS Command Line Interface

This section explains how to configure settings that the AWS Command Line Interface uses when interacting with AWS, such as your security credentials and the default region.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: ENTER
To use this example, type aws configure at the command line and press Enter. aws configure is the command. This command is interactive, so the AWS CLI outputs lines of texts, prompting you to enter additional information. Enter each of your access keys in turn and press Enter. Then, enter a region name in the format shown, press Enter, and press Enter a final time to skip the output format setting. The final Enter command is shown as replaceable text because there is no user input for that line

Configuration and Credential Files

The CLI stores credentials specified with aws configure in a local file named credentials in a folder named .aws in your home directory. Home directory location varies but can be referred to using the environment variables %UserProfile% in Windows and $HOME or ~ (tilde) in Unix-like systems.

For example, the following commands list the contents of the .aws folder:

Linux, OS X, or Unix

$ ls  ~/.aws
The files generated by the CLI for the profile configured in the previous section look like this:

~/.aws/credentials

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
~/.aws/config

[default]
region=ap-southeast-1a
output=json

AWS Command line is installed and configured, you can check by typing
$aws help