Practical Infrastructure automation using Python and Ansible

Table of Contents

Ansible is an open source IT Configuration Management, Deployment & Orchestration tool. It aims to provide large productivity gains to a wide variety of automation challenges. This tool is very simple to use yet powerful enough to automate complex multi-tier IT application environments.

Why Ansible?

Its Next Generation DeOps configuration management tool easy to write, read, maintain and evolve- without writing Scripts or custom code, Fast to learn and setup

It uses a very simple language (YAML, in the form of Ansible Playbooks) that allows to describe the automation jobs in a way that approaches plain English. Doesn’t require a custom agent or software to install, Ansible works by connecting to your nodes and pushing out small programs, called "Ansible modules" to them.

How Ansible is different?

You might be wondering how Ansible is different from other configuration management tools, like Puppet and Chef for example, so I thought it would be useful to mention a couple items, before we move onto more complex examples. Probably the most glaring difference is that Ansible pushes the configuration out to each managed machine via ssh. Ansible only requires that we install the Ansible software onto a management node, and that the remote machines is running ssh with python installed, which every major distribution come with by default, there are no remote agents that we need to install, and everything is done via ssh from the management node.

Ansible, that has over 250 helper modules, or functions included with Ansible. These allow you to construct playbooks to smartly add users, install ssh keys, tweak permissions, deploy code, install packages, interact with cloud providers for things like launch instances or modifying a load balancer, etc. Each module has a dedicated page on the Ansible documentation site, along with detailed examples, and I have found this a major bonus to working with Ansible. I should mention that, even though Ansible is using ssh to connect to these remote machines, playbooks and ad-hoc commands will almost always be using these 250 plus modules to smartly do things.

Installing Ansible (How to build a Central Server and Nodes)

Insert CentOS install disk and start the Computer or Server. Push Enter key to proceed.

Select the installation language.

This is the default screen for some basic configuration. First, set time zone. Click the "DATE & TIME" icon.

Click a point on the map you'd like to set as your time zone and press "Done" button which is on the upper left.

It will go back to the default screen. Click the "KEYBOARD" icon next.

You can click "+" button on left to add another keyboard for your language.

Select your required keyboard type and click "Add" button.

Back to the default screen. For "SOFTWARE SELECTION", it's OK to keep default because it had better to install with "Minimal Install" for initial installation. Next, click the "INSTALLATION DESTINATION" icon.

Select installation disk. Click the disk icon you'd like to install and next, click "Done" button on upper left to proceed. If you'd like to customize partition layouts, check box "I will configure partitioning" and proceed.

Back to the default screen. Click the "NETWORK & HOSTNAME" icon.

Input any hostname you like in "Hostname" field and click "ON" button on upper-right corner to enable networking.

Click "Done" button to finish.

After verifying the installation summary, click "Begin Installation".

Installation will start and we need to set a root password and create a common user. Click each icon and move to configuration.

On the root password settings, input any password you like and click "Done" button to finish as follows.

On the common user settings, input any user name and password you like and click "Done" button to finish.

After finishing installation, click "Reboot" button.

After rebooting, login with root user and password you set during the installation. Installing CentOS is completed if you can login successfully.

Setup additional repository for CentOS 7

Add Repositories

Add some useful external repositories to install useful software.

Install a plugin to add priorities to each installed repository.

yum -y install yum-plugin-priorities

set [priority=1] to official repository

sed -i -e "s/\]$/\]\npriority=1/g" /etc/yum.repos.d/CentOS-Base.repo

Add EPEL Repository which is provided from Fedora project.

yum -y install epel-release

set [priority=5]

sed -i -e "s/\]$/\]\npriority=5/g" /etc/yum.repos.d/epel.repo

for another way, change to [enabled=0] and use it only when needed

sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/epel.repo

if [enabled=0], input a command to use the repository

yum --enablerepo=epel install [Package]

Adding CentOS SCLo Software collections Repository

yum -y install centos-release-scl-rh centos-release-scl

# set [priority=10]

sed -i -e "s/\]$/\]\npriority=10/g" /etc/yum.repos.d/CentOS-SCLo-scl.repo

sed -i -e "s/\]$/\]\npriority=10/g" /etc/yum.repos.d/CentOS-SCLo-scl-rh.repo

# for another way, change to [enabled=0] and use it only when needed

sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-SCLo-scl.repo

sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/CentOS-SCLo-scl-rh.repo

if [enabled=0], input a command to use the repository

yum --enablerepo=centos-sclo-rh install [Package]

yum --enablerepo=centos-sclo-sclo install [Package]

Add Remi's RPM Repository which provides some useful packages

yum -y install http://rpms.famillecollet.com/enterprise/remi-release-7.rpm

set [priority=10]

sed -i -e "s/\]$/\]\npriority=10/g" /etc/yum.repos.d/remi-safe.repo

for another way, change to [enabled=0] and use it only when needed

sed -i -e "s/enabled=1/enabled=0/g" /etc/yum.repos.d/remi-safe.repo

if [enabled=0], input a command to use the repository

yum --enablerepo=remi-safe install [Package]

Server Harding & security Best Practices

Create a Standard User Account

For security reasons, it is not advisable to be performing daily computing tasks using the root account. Instead, it is recommended to create a standard user account that will be using sudo to gain administrative privileges. To create a user account named Nuhan, type:

adduser Nuahn

Set a password for the new user. You'll be prompted to input and confirm a password.

passwd Nuhan

useradd Nuahan && passwd Nuhan (Single command to setup user account)

Add the new user to the wheel group so that it can assume root privileges using sudo.

gpasswd -a Nuhan wheel

Disallow Root Login and Password Authentication

sudo vi /etc/ssh/sshd_config

Look for the PermitRootLogin line, uncomment it and set the value to no.

PermitRootLogin no

Do the same for the PasswordAuthentication line, which should be uncommented already:

PasswordAuthentication no

Save and close the file. To apply the new settings, reload SSH.

Enable the IP Tables Firewall

By default, the active firewall application on a newly activated CentOS 7 server is FirewallD.

Many security applications still do not have support for it. So if you are going to use any applications like OSSEC HIDS, it's best to disable/uninstall FirewallD.

sudo yum remove -y firewalld

Install/activate IPTables.

sudo yum install -y iptables-services

sudo systemctl start iptables

Configure IPTables to start automatically at boot time.

sudo systemctl enable iptables

IPTables on CentOS 7 comes with a default set of rules, which we can view with the following command.

sudo iptables -L -n

Those rules are runtime rules and will be lost on reboot, it's best to save them to a file using:

sudo /usr/libexec/iptables/iptables.init save

That command will save the rules to the /etc/sysconfig/iptables file. We can edit the rules anytime by changing this file.

Allow Additional Traffic through the Firewall

In order for the server to host websites or applications, we have to add new rules to the firewall to allow HTTP and HTTPS traffic. To accomplish that, open the IPTables file,

sudo nano /etc/sysconfig/iptables

Just after or before the SSH rule, add the rules for HTTP (port 80) and HTTPS (port 443) traffic, so that that portion of the file appears as shown in the code block below. This Iptable rules can be setup as per the requirement.

-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT

-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

Save and close the file, then reload IPTables.

sudo systemctl reload iptables

With the above step completed, CentOS 7 server should now be reasonably secure and be ready for use in production. After Initial Setup CentOS, Update the system if possible.

Ansible Installation & Validation

Control Machine System Requirements

Currently Ansible can be run from any machine with Python 2.6 or 2.7 installed (Windows isn’t supported for the control machine).

Node Machine System Requirements

On the managed nodes, we need a way to communicate, which is normally ssh. By default this uses sftp. If that’s not available, we can switch to scp in ansible.cfg. We also need Python 2.4 or later, if the system is running less than Python 2.5 on the remotes. We also need python-simplejson.

Install the epel-release RPM if needed on CentOS, RHEL, or Scientific Linux

$ sudo yum install ansible

Latest Release via Yum

$ sudo apt-get install software-properties-common

$ sudo apt-add-repository ppa:ansible/ansible

$ sudo apt-get update

$ sudo apt-get install ansible

sudo yum -y install ansible

Latest Release via Pip

$ sudo easy install pip

$ sudo pip install ansible

Ansible is installed (that simple!)

Validate Ansible Installation

Ansible –version

Ansible Components

Controller Machine: The machine where Ansible is installed, responsible for running the provisioning on the servers we are managing.
Inventory: An initialization file that contains information about the servers we are managing.
Playbook: The entry point for Ansible provisioning, where the automation is defined through tasks using YAML format.
Task: A block that defines a single procedure to be executed, e.g. Install a package.
Module: A module typically abstracts a system task, like dealing with packages or creating and changing files. Ansible has a multitude of built-in modules, but we can also create custom ones.
Role: A pre-defined way for organizing playbooks and other files in order to facilitate sharing and reusing portions of a provisioning.
Play: A provisioning executed from start to finish is called a play. In simple words, execution of a playbook is called a play.
Facts: Global variables containing information about the system, like network interfaces or operating system.

Ad-Hoc Commands on Inventory

Runs some shell command on the hosts and Groups you define

$ ansible –m shell –a ‘ls-al’ web01

$ ansible –m shell –a ‘whomai’ web01

$ ansible –m shell –a ‘ifconfig’ webservers

$ ansible –m shell –a ‘hostmae’ dc

Tips

A pattern usually refers to a set of groups (which are sets of hosts)

$ ansible -m ping all

$ ansible -m ping web*

$ ansible -m ping 'appservers:dbservers'

$ ansible -m ping 'dc:!webservers'

$ ansible -m ping 'dc:&webservers'

Tasks as Ad-Hoc Commands

Ansible can execute single tasks on sets of hosts to full-fill an

ad-hoc declarations.

$ ansible webservers -m file -a "path=/var/www/html/assets state=directory"

$ ansible webservers -m apt -a "name=nginx state=present"

$ ansible webservers -m service -a "name=nginx enabled=yes state=started"

Modules Documentation

Module listing and documentation via ansible-doc

$ ansible-doc -l

$ ansible-doc apt

Module index

http://docs.ansible.com/ansible/modules_by_category.html

Install Nginx with Ad-Hoc Commands

Install the nginx server on webservers group with yum module

$ ansible -m yum -a "name=nginx state=present update_cache=yes" web01

Ensure service enabled and started on webservers with service module

$ ansible –m service –a “name=nginx sate=started enabled=yes” webservers –become

Ensure /usr/share/nginx/html directory exists on web servers with file module

$ ansible –m file –a “path=/usr/share/nginx/html state=directory” webservers --become

Update /usr/share/nginx/html/index.html file a custom file with copy module

$ ansible -m copy -a "src=index.html dest=/usr/share/nginx/html/index.html"

webservers –become

Become (Privilege Escalation)

Ansible can use existing privilege escalation systems to allow a user to execute tasks as another.

Ansible allows to ‘become’ another user, different from the user that logged into the machine (remote user). This is done using existing privilege escalation tools, which we already used or have configured, like sudo, su, pfexec, doas, pbrun, dzdo, ksu and others.

$ ansible -m shell -a "whoami" web01 --become

Ansible Playbook Writing

Real power of Ansible lies in it’s scripting capabilities using playbooks.

Puppet => Module

Chef => Cookbook

Use playbooks to perfom many actions on multiple machines. Playbooks are written in YAML.

Play Books are divided into 3 sections

· Traget section - Defines on which host or hosts this play book will be executed and how it will be executed

· Variables section – Defines variables which can be used from playbooks.

· Tasks Section – List all the modules that we intend to run in the order.

Lab Setup 01 Details

Deployment Objectives	Task
Common (Paly 01)	Disable selinux Create a standard directory Install vim
Webserver	Install apache webserver Create webadmin user
Database	Install mariadb database server Create dbadmin user

Deployment I – Play 1

Create the file name as follows

vi common.yml

Add these content in to above file and execute ansible playbook complete the paly

ansible-playbook common.yml –check

ansible-playbook common.yml --check

----

- hosts: all

tasks:

- name: Disable SELinux

selinux:

state: disabled

- name: Create MyFiles Directory

file:

path: /root/MyFiles

state: directory

owner: root

group: root

mode: 0755

- name: Install Vim

yum:

name: vim

state: present

Deployment I – Play 2

Create the file name as follows

vi webserver.yml

Add these content in to above file and execute ansible playbook complete the paly

ansible-playbook webserver.yml –check

ansible-playbook webserver.yml

---

- hosts: webservers

tasks:

- name: Install Apache Webserver

yum:

name: httpd

state: present

- name: Start Apache Webserver

service:

name: httpd

state: started

enabled: yes

- name: Create webadmin user

user:

name: webadmin

comment: "Web Admin User"

groups: apache

Deployment I – Play 3

Create the file name as follows

vi dbservers.yml

Add these content in to above file and execute ansible playbook complete the paly

ansible-playbook dbserver.yml –check

ansible-playbook dbserver.yml

---

- hosts: dbservers

tasks:

- name: Install MariaDB Server

yum:

name: mariadb-server

state: present

- name: Start MariaDB Server

service:

name: mariadb-server

state: started

enabled: yes

- name: Create dbadmin user

user:

name: dbadmin

comment: "DB Admin User"

groups: mysql

Advance Infrastructure automation using Python and Ansible

In this section, we are going create a load balanced web cluster using Vagrant and Ansible. The goal is to show how Ansible can solve real problems by building up complex infrastructure from scratch.

What We Are Building

The end result of our activities today, will be a Vagrant environment with a fully functioning haproxy load balancer, with six nginx web servers sitting behind it. We will use Ansible to install all of the required packages, deploy configuration files, and start the correct services on to each of these boxes. All without logging into any of these nodes manually.

We will then fire up a web browser and start sending requests to the load balancer. We will watch, as our requests cycle through the six web servers, while we hit refresh in the browser. This is not only a pretty cool demo, but we will also learn some new things about Ansible, as we work to get this configured.

Deployment Objectives

Task

Setup Lab Environment

Install Virtual box

sudo apt-get install virtualbox

Install Rubbygem

sudo apt-get install rubygems-integration

Install Vagrant

sudo apt-get install vagrant

sudo apt-get install virtualbox-dkms

Create ample Vagrant file

vagrant init

Setup Vagrant file

Setup bootstrap-mgmt.sh

Setup ansible configuration file

vi /ansible.cfg

[defaults]

hostfile = /home/vagrant/inventory.ini

Setup inventory.ini file

cd /home/vagrant

pwd

vi inventory.ini

[lb]

[web]

web1

web2

web3

web4

web5

web 6

Verifying Network Connectivity

We should verify our management node has connectivity to all web servers and Load Balancer.

Configuring SSH Access

ssh-keyscan web1 web2 web3 web4 web4 web6 >> .ssh/known_hosts

setup ssh-addkey playbook

With that done, lets push out the vagrant users public RSA key, to web3 web4 web5 web6, through the use of the ssh-addkey playbook. Let’s run, ansible-playbook ssh-addkey, and since this is the first time we are connection to our new web nodes, we need to tack on the ask pass option, since we do not have password less access configured yet. The password is just vagrant.

ansible-playbook
  ssh-addkey.yml --ask-pass

Verify Ansible Connectivity

Now that we have the hosts inventory updated, and pushed out our ssh key, lets verify Ansible can talk to all nodes

ansible
  all -m ping

vagrant@mgmt:~$ ansible all -m ping

web1 | success >> {

"changed": false,

"ping": "pong"

}

web2 | success >> {

"changed": false,

"ping": "pong"

}

We get back ping pong messages from all the hosts like above in our environment. Cool, so this proves that we have connectivity, and that things are in good shape.

Setup site.yml playbook

Setup Template Directory on Ansible Controller.

mkdir templates

default-site

haproxy.cfg

index.html

nginx.conf

Show host name using ansible factor

ansible web1 –m setup –a “filter=ansible_hostname”

Show web1 network realted information

ansible web1 –m setup –a “filter=ansible_eth1”

Install Loadbalancer node

ansible –playbook site.yml

Testing

localhost:8080/haproxy? Stats

curl –I http://localhost:8080/

ab –n 10000 –c 25 http://localhsot:8080

Setup Vagrant file

Setup Vagrantfile on vagrant Project Directory as per our requirement.

vi Vagrantfile

# Defines our Vagrant environment

# -*- mode: ruby -*-

# vi: set ft=ruby :

Vagrant.configure("2") do |config|

# create mgmt node

config.vm.define :mgmt do |mgmt_config|

mgmt_config.vm.box = "ubuntu/trusty64"

mgmt_config.vm.hostname = "mgmt"

mgmt_config.vm.network :private_network, ip: "10.0.15.10"

mgmt_config.vm.provider "virtualbox" do |vb|

vb.memory = "256"

end

mgmt_config.vm.provision :shell, path: "bootstrap-mgmt.sh"

end

# create load balancer

config.vm.define :lb do |lb_config|

lb_config.vm.box = "ubuntu/trusty64"

lb_config.vm.hostname = "lb"

lb_config.vm.network :private_network, ip: "10.0.15.11"

lb_config.vm.network "forwarded_port", guest: 80, host: 8080

lb_config.vm.provider "virtualbox" do |vb|

vb.memory = "256"

end

# create web servers

(1..6).each do |i|

config.vm.define "web#{i}" do |node|

node.vm.box = "ubuntu/trusty64"

node.vm.hostname = "web#{i}"

node.vm.network :private_network, ip: "10.0.15.2#{i}"

node.vm.network "forwarded_port", guest: 80, host: "808#{i}"

node.vm.provider "virtualbox" do |vb|

vb.memory = "256"

end

Setup bootstrap-mgmt.sh

Setup Vagrantfile on vagrant Project Directory as per our requirement.

vi bootstrap-mgmt.sh

#!/usr/bin/env bash

# install ansible (http://docs.ansible.com/intro_installation.html)

apt-get -y install software-properties-common

apt-add-repository -y ppa:ansible/ansible

apt-get update

apt-get -y install ansible

# copy examples into /home/vagrant (from inside the mgmt node)

cp -a /vagrant/examples/* /home/vagrant

chown -R vagrant:vagrant /home/vagrant

# configure hosts file for our internal network defined by Vagrantfile

cat >> /etc/hosts <<EOL

# vagrant environment nodes

10.0.15.10 mgmt

10.0.15.11 lb

10.0.15.21 web1

10.0.15.22 web2

10.0.15.23 web3

10.0.15.24 web4

10.0.15.25 web5

10.0.15.26 web6

EOL

chmod +x bootstrap-mgmt.sh

Run vagrant up command to bring all machine (lab environment up and running)

vagrant up

vagrant status (To see the virtual machine status)

Setup ssh-addkey.yml playbook

---

- hosts: all

sudo: yes

gather_facts: no

remote_user: vagrant

tasks:

- name: install ssh key

authorized_key: user=vagrant

key="{{ lookup('file', '/home/vagrant/.ssh/id_rsa.pub') }}"

state=present

Setup site.yml playbook (Setup Loadbalancer & Wolfram)

---

# common

- hosts: all

sudo: yes

gather_facts: no

tasks:

- name: install git

apt: name=git state=installed update_cache=yes

# web

- hosts: web

sudo: yes

tasks:

- name: install nginx

apt: name=nginx state=installed

- name: write our nginx.conf

notify: restart nginx

- name: write our /etc/nginx/sites-available/default

notify: restart nginx

- name: deploy website content

handlers:

- name: restart nginx

service: name=nginx state=restarted

# lb

- hosts: lb

sudo: yes

tasks:

- name: install haproxy and socat

apt: pkg={{ item }} state=latest

with_items:

- haproxy

- socat

- name: enable haproxy

lineinfile: dest=/etc/default/haproxy regexp="^ENABLED" line="ENABLED=1"

notify: restart haproxy

- name: deploy haproxy config

notify: restart haproxy

handlers:

- name: restart haproxy

service: name=haproxy state=restarted

Setup default-site template configuration file

default-site

# {{ ansible_managed }}

server {

listen 80;

server_name {{ ansible_hostname }};

root /usr/share/nginx/html;

index index.html index.htm;

location / {

try_files $uri $uri/ =404;

}

error_page 404 /404.html;

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root /usr/share/nginx/html;

}

Setup haproxy.cfg template configuration file

# {{ ansible_managed }}

global

log 127.0.0.1 local2

chroot /var/lib/haproxy

pidfile /var/run/haproxy.pid

maxconn 4000

user root

group root

daemon

# turn on stats unix socket

stats socket /var/lib/haproxy/stats level admin

defaults

mode http

log global

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

retries 3

timeout http-request 10s

timeout queue 1m

timeout connect 10s

timeout client 1m

timeout server 1m

timeout http-keep-alive 10s

timeout check 10s

maxconn 3000

# enable stats uri

stats enable

stats uri /haproxy?stats

backend app

{% for host in groups['lb'] %}

listen episode46 {{ hostvars[host]['ansible_eth0']['ipv4']['address'] }}:80

{% endfor %}

balance roundrobin

{% for host in groups['web'] %}

server {{ host }} {{ hostvars[host]['ansible_eth1']['ipv4']['address'] }} check port 80

{% endfor %}

Setup index.html template configuration file

<html>

<title># Ansible Lab</title>

<body>

<div class="block" style="height: 99%;">

<div class="centered">

<h1>#46 Demo</h1>

<p>Served by {{ ansible_hostname }} ({{ ansible_eth1.ipv4.address }}).</p>

</div>

</body>

</html>

Setup nginx.conf template configuration file

# {{ ansible_managed }}

user www-data;

worker_processes 1;

pid /var/run/nginx.pid;

worker_rlimit_nofile 1024;

events {

worker_connections 512;

}

http {

include /etc/nginx/mime.types;

default_type application/octet-stream;

tcp_nopush "on";

tcp_nodelay "on";

#keepalive_timeout "65";

access_log "/var/log/nginx/access.log";

error_log "/var/log/nginx/error.log";

server_tokens off;

types_hash_max_size 2048;

# https://philio.me/backend-server-host-name-as-a-custom-header-with-nginx/

add_header X-Backend-Server $hostname;

# disable cache used for testing

add_header Cache-Control private;

add_header Last-Modified "";

sendfile off;

expires off;

etag off;

include /etc/nginx/conf.d/*.conf;

include /etc/nginx/sites-enabled/*;

}

Tuesday, November 7, 2017

Practical Infrastructure automation using Python and Ansible

No comments:

Post a Comment