Chris Reeves' Blog | Just another IT Blog

Automate SSH Key Rotation on Ubuntu with Ansible

Posted 15/11/2018 by Chris & filed under Ansible, Hardening, Linux & Unix.

Overview

Changing your SSH keys is as important as changing your underpants daily, running this script on a frequent basis will ensure access to the servers are changed on a regular basis. Use Ansible to do ssh key rotation in your sleep!

Test Bed

Ansible control server running Ubuntu 18.04 LTS
Test server running Ubuntu 18.04 LTS

Requirements

Ansible control server
SSH keys established between Ansible control server and destination server(s)
A folder called “pubkeys” where the script is running from

Break Down

Creates a new directory on the remote server to generate the new keys on
Generates the new key pair in the newly formed folder
Copies the new public key to the local machine running the ansible script under /pubkeys/ and names it “id_rsa.%hostname%.pub
Removes existing private key
Removes existing public key
Moves new private key to the users .ssh folder
Moves new public key to the users .ssh folder
Changes new private key to read only
Invalidates existing keys and applies the public key copied to the local host to the server
Copies the new private key local host and changes the file to “id_rsa.%hostname%
Removes “newsshkey” folder on remote host as a clean up

ssh_key_rotation.yml

---
- hosts: test
 
  tasks:
 
  - name: Creates directory
    file:
      path: ~/.ssh/newsshkey
      state: directory
 
  - name: Create New SSH Key Pair
    command: ssh-keygen -f id_rsa -t rsa -N '' -f ~/.ssh/newsshkey/id_rsa
 
  - fetch:
      src: ~/.ssh/newsshkey/id_rsa.pub
      dest: pubkeys/{{ inventory_hostname }}.pub
      flat: yes
 
  - name: Remove Old Keys
    command: rm -rf ~/.ssh/id_rsa
 
  - name: Remove Old Pub Key
    command: rm -rf ~/.ssh/id_rsa.pub
 
  - name: Move New Key
    command: mv ~/.ssh/newsshkey/id_rsa ~/.ssh/
 
  - name: Move New Pub Keys
    command: mv ~/.ssh/newsshkey/id_rsa.pub ~/.ssh/
 
  - name: Change id_rsa Permissions
    command: chmod 400 ~/.ssh/id_rsa
 
  - name: Set authorized key, removing all the authorized key already set
    authorized_key:
      user: root
      key: '{{ item }}'
      state: present
      exclusive: True
    with_file:
      - pubkeys/{{ inventory_hostname }}.pub
 
  - fetch:
      src: ~/.ssh/id_rsa
      dest: ~/.ssh/id_rsa.{{ inventory_hostname }}
      flat: yes
 
  - name: Removing existing folder
    file:
      path: ~/.ssh/newsshkey
      state: absent

---

- hosts: test

tasks:

- name: Creates directory

file:

path: ~/.ssh/newsshkey

state: directory

- name: Create New SSH Key Pair

command: ssh-keygen -f id_rsa -t rsa -N '' -f ~/.ssh/newsshkey/id_rsa

- fetch:

src: ~/.ssh/newsshkey/id_rsa.pub

dest: pubkeys/{{ inventory_hostname }}.pub

flat: yes

- name: Remove Old Keys

command: rm -rf ~/.ssh/id_rsa

- name: Remove Old Pub Key

command: rm -rf ~/.ssh/id_rsa.pub

- name: Move New Key

command: mv ~/.ssh/newsshkey/id_rsa ~/.ssh/

- name: Move New Pub Keys

command: mv ~/.ssh/newsshkey/id_rsa.pub ~/.ssh/

- name: Change id_rsa Permissions

command: chmod 400 ~/.ssh/id_rsa

- name: Set authorized key, removing all the authorized key already set

authorized_key:

user: root

key: '{{ item }}'

state: present

exclusive: True

with_file:

- pubkeys/{{ inventory_hostname }}.pub

- fetch:

src: ~/.ssh/id_rsa

dest: ~/.ssh/id_rsa.{{ inventory_hostname }}

flat: yes

- name: Removing existing folder

file:

path: ~/.ssh/newsshkey

state: absent

Note: You will need to change/remove the “- hosts:” entry

Key Management

To fully automate this I have mounted a cifs share and created a symbolic link on the Ansible server from the ~/.ssh folder to the cifs share. All my other clients are set up the same way so when you update the key it copies the key to a central repository which all other clients are symbolically linked to.

Conclusion

This can be greatly be improved on but is a good starting point in the rotation of your ssh keys. I’m happy to hear suggestions on how this could be improved.

Ansible – mySQL root password change on Ubuntu

Posted 24/10/2018 by Chris & filed under Ansible, Linux & Unix, mySQL.

This Ansible script will fully rotate your MySQL root account passwords (or change any MySQL account passwords if you change the script) and implement my.cf so you don’t have to keep putting the password in. This took me a while to figure out, there are

Test Bed

Ansible control server running Ubuntu 18.04
Ubuntu 18.04 Bionic test server running mySQL 5.7.25

Requirements

Ansible control server
SSH keys established between Ansible control server and destination server

Overview

Install mySQL package with required dependancies
Stop mySQL service
Set mySQL environment variables
Start mySQL
Change mySQL root password to a mySQL native password (native is very important!)
Copy .my.cnf from local source to ~
mySQL flush privileges
Stop mySQL service
Unset mySQL environment variables
Start mySQL

sql.yml

---
  - hosts: test

    var_files:
    - global_vars/config.yml
   
    - name: Install mySQL Server
      apt: name=mysql-server state=present
    - name: Install Additional Dependencies
      apt:
        pkg:
          - postfix-mysql
          - dovecot-imapd
          - dovecot-lmtpd
          - dovecot-mysql
          - python-pymysql
          - mysql-client
          - mysql-common
        state: present
# MySQL 
    - name: Ensure mySQL is running and starts on boot
      service: name=mysql state=started enabled=true

    - name: Stop MySQL
      service: name=mysql state=stopped
    - name: set environment variables
      shell: systemctl set-environment MYSQLD_OPTS="--skip-grant-tables"
    - name: Start MySQL
      service: name=mysql state=started
    - name: sql query
      command:  mysql -u root --execute="ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '"{{ mysql_root_password }}"';"
    - name: Create ~/.my.cnf
      file:
        path: ~/.my.cnf
        state: touch
    - name: Insert into ~/.my.cnf
      blockinfile:
        path: ~/.my.cnf
        block: |
          [client]
          user={{ mysql_user }}
          password={{ mysql_root_password }}
    - name: sql query flush
      command:  mysql -u root --execute="FLUSH PRIVILEGES"
    - name: Stop MySQL
      service: name=mysql state=stopped
    - name: unset environment variables
      shell: systemctl unset-environment MYSQLD_OPTS
    - name: Start MySQL
      service: name=mysql state=started

---

- hosts: test

var_files:

- global_vars/config.yml

- name: Install mySQL Server

apt: name=mysql-server state=present

- name: Install Additional Dependencies

apt:

pkg:

- postfix-mysql

- dovecot-imapd

- dovecot-lmtpd

- dovecot-mysql

- python-pymysql

- mysql-client

- mysql-common

state: present

# MySQL

- name: Ensure mySQL is running and starts on boot

service: name=mysql state=started enabled=true

- name: Stop MySQL

service: name=mysql state=stopped

- name: set environment variables

shell: systemctl set-environment MYSQLD_OPTS="--skip-grant-tables"

- name: Start MySQL

service: name=mysql state=started

- name: sql query

command: mysql -u root --execute="ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '"{{ mysql_root_password }}"';"

- name: Create ~/.my.cnf

file:

path: ~/.my.cnf

state: touch

- name: Insert into ~/.my.cnf

blockinfile:

path: ~/.my.cnf

block: |

[client]

user={{ mysql_user }}

password={{ mysql_root_password }}

- name: sql query flush

command: mysql -u root --execute="FLUSH PRIVILEGES"

- name: Stop MySQL

service: name=mysql state=stopped

- name: unset environment variables

shell: systemctl unset-environment MYSQLD_OPTS

- name: Start MySQL

service: name=mysql state=started

etc/mysql/.my.cf

[client]
user=root
password={{ mysql_root_password }}

[client]

user=root

password={{ mysql_root_password }}

global-vars/config.yml

mail_db_host: localhost
mysql_user: root
mysql_root_password: changepwhere

mail_db_host: localhost

mysql_user: root

mysql_root_password: changepwhere

Execution

ansible-playbook sql.yml

1	ansible-playbook sql.yml

Other Considerations

You will need to remove lines 7 to 19 if you are not installing MySQL for the first time.

If any applications are using the account you are rotating, the application will auth fail (I would hope your not using root for app authentication) – if you use this against any other username this will need to be considered.

Conclusion

The gotcha for a lot of people (from what I’ve read on blogs/github) is that when the mysql root password changes you also need to change it from “auth_socket” to “mysql_native_password”.

Regenerating SID’s using Powershell

Posted 14/08/2018 by Chris & filed under Microsoft, Powershell.

I’ve devised a handy way of regenerating SID’s on Microsoft operating systems using a third party tool and a little PowerShell magic.

The Breakdown

The first part of the script sets variables that can be called upon to randomise the SID number.

The next part sets the source location to download newsid.exe and defines a destination location.

The “set-itemproperty” adds a key into the registry to trick Newsid to think it has already accepted the EULA.

The “start-bitstransfer” initiates the download of newsid.

The real magic is in the last part – newsid is executed with “/a” to run with no prompts and “/n” to not reboot after.

execution”. The “S-1-5-21-“ is the first part of any SID then it calls the three random numbers generated from $random%.

The Script

$random1 = Get-Random 10000000000
$random2 = Get-Random 10000000000
$random3 = Get-Random 10000000000

$sidsource = "https://source.domain.com/newsid.exe"
$siddest = "C:\Scripts\newsid.exe"

Set-ItemProperty -Path "HKCU:\Software\Sysinternals\NewSID" -Name "EulaAccepted" -Value 1
Start-BitsTransfer -Source $sidsource -Destination $siddest

C:\Scripts\newsid.exe /a /n S-1-5-21-$random1-$random2-$random3

$random1 = Get-Random 10000000000

$random2 = Get-Random 10000000000

$random3 = Get-Random 10000000000

$sidsource = "https://source.domain.com/newsid.exe"

$siddest = "C:\Scripts\newsid.exe"

Set-ItemProperty -Path "HKCU:\Software\Sysinternals\NewSID" -Name "EulaAccepted" -Value 1

Start-BitsTransfer -Source $sidsource -Destination $siddest

C:\Scripts\newsid.exe /a /n S-1-5-21-$random1-$random2-$random3

Considerations

Having this script on a gold imaged server set to auto run on login (with a restart afterwards) would be the best implementation of this. You wouldn’t need to use the source and destination either and just point to a local copy of newsid.exe. I did however write this with a remote copy of newsid.exe in mind.

You will need to change the source and destination to suit your needs.

This is written for Microsoft PowerShell Version 5+.

You could drop the “/n” in the newsid.exe switch so it does reboot on completion.

The “S-1-5-21-” at the beginning of the SID is always going to be the same.

Conclusion

With the ever increasing demand for automating the deployment of servers, this is as important as ever if you start interconnecting the automated servers.

Atlassian Backup to AWS S3

Posted 21/06/2018 by Chris & filed under Amazon Web Services, Atlassian.

If you have a self hosted Atlassian suite (Bitbucket, Confluence or Jira) you might want to consider creating a backup of the data to an external repository such as AWS S3 in-case your server fails.

Overview

This script creates a tar of all your Atlassian data and uploads it to AWS s3 using the aws cli package.

Requirements

1 x AWS S3 bucket
AWS programmatic account that has writeable access to the AWS S3 bucket
AWS CLI installed on the Linux server and configured with a AWS programmatic account

Script

#!/bin/bash

host=atlassian
bucket=yourbucketnamegoeshere
tmp=/tmp/backup.cache
date=(`date +%d-%m-%Y`)
year=(`date +%C%g`)
month=(`date +%B`)
day=(`date +%d`)

#Backup Bitbucket Repositories
echo Backing up Bitbucket Repositories

#Tar up data
tar cvzfP $tmp/atlassian-repos-$date.tar.gz /var/atlassian/application-data/bitbucket/shared/data/repositories

#Transfer tar to AWS S3
aws s3 cp $tmp/atlassian-repos-$date.tar.gz s3://$bucket/$year/$month/$day/$host/atlassian-repos-$date.tar.gz

#Remove tar locally
rm -rf $tmp

#!/bin/bash

host=atlassian

bucket=yourbucketnamegoeshere

tmp=/tmp/backup.cache

date=(`date +%d-%m-%Y`)

year=(`date +%C%g`)

month=(`date +%B`)

day=(`date +%d`)

#Backup Bitbucket Repositories

echo Backing up Bitbucket Repositories

#Tar up data

tar cvzfP $tmp/atlassian-repos-$date.tar.gz /var/atlassian/application-data/bitbucket/shared/data/repositories

#Transfer tar to AWS S3

aws s3 cp $tmp/atlassian-repos-$date.tar.gz s3://$bucket/$year/$month/$day/$host/atlassian-repos-$date.tar.gz

#Remove tar locally

rm -rf $tmp

Considerations

You will need to change lines 3 and 4 to match your requirements.

By default data is located in the “/var/atlassian/appliction-data” however you will need to change this location if it isn’t.

It creates the following folder structure in the S3 bucket: /year/month/day/host

Add this to a cron for automated results.

Conclusion

By setting up this script to run in a cron you will have the data in at least two places and depending on your cron schedule a lot of points to restore from.

Nagios XI – Adding Microsoft Server Services via API

Posted 28/04/2018 by Chris & filed under Linux & Unix, Nagios.

When automating server provisioning you might be considering implementing Nagios for monitoring servers and services as part of the build process. Apart from the help section of Nagios console giving some hints on how to interact with the API, there isn’t much else to go on so I’m sharing my learnings on how to do this.

I’m not going to go through the whole implementation of automating Nagios into server builds but more specifically connecting to the API.

The Nagios reference is http://host.example.com/nagiosxi/help/ and then click on “Config Reference”

The first step is to add the host – now this is in the Nagios help section so hopefully, this is nothing new

curl -XPOST &quot;http://nagios.example.com/nagiosxi/api/v1/config/host?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&amp;pretty=1&quot; -d &quot;host_name=testapihost&amp;address=127.0.0.1&amp;check_command=check_ping\!3000,80%\!5000,100%&amp;max_check_attempts=2&amp;check_period=24x7&amp;contacts=nagiosadmin&amp;notification_interval=5&amp;notification_period=24x7&amp;applyconfig=1&quot;

curl -XPOST "http://nagios.example.com/nagiosxi/api/v1/config/host?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&pretty=1" -d "host_name=testapihost&address=127.0.0.1&check_command=check_ping\!3000,80%\!5000,100%&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"

You will need to change the FQDN, API Key, Hostname and address to your requirements to make this work for you.

Scroll to the right of the code to see it all.

One thing to note is every setting is separated by a “&” and variables specified by the check start with “!”. This is the key to understanding how this works.

Add Processor

curl -XPOST &quot;http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&amp;pretty=1&quot; -d &quot;host_name=testapihost&amp;service_description=CPU&amp;check_command=check_xi_service_nsclient\!password\!CPULOAD\!80,90&amp;check_interval=5&amp;retry_interval=5&amp;max_check_attempts=2&amp;check_period=24x7&amp;contacts=nagiosadmin&amp;notification_interval=5&amp;notification_period=24x7&amp;applyconfig=1&quot;

curl -XPOST "http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&pretty=1" -d "host_name=testapihost&service_description=CPU&check_command=check_xi_service_nsclient\!password\!CPULOAD\!80,90&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"

Add Disk

curl -XPOST &quot;http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&amp;pretty=1&quot; -d &quot;host_name=testapikey&amp;service_description=CDisk&amp;check_command=check_xi_service_nsclient\!password\!USEDDISKSPACE\!-l C -w 80 -c 95&amp;check_interval=5&amp;retry_interval=5&amp;max_check_attempts=2&amp;check_period=24x7&amp;contacts=nagiosadmin&amp;notification_interval=5&amp;notification_period=24x7&amp;applyconfig=1&quot;

curl -XPOST "http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&pretty=1" -d "host_name=testapikey&service_description=CDisk&check_command=check_xi_service_nsclient\!password\!USEDDISKSPACE\!-l C -w 80 -c 95&check_interval=5&retry_interval=5&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"

Add Memory

curl -XPOST &quot;http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&amp;pretty=1&quot; -d &quot;host_name=testapihost&amp;service_description=Mem&amp;check_command=check_xi_service_nsclient\!password\!MEMUSE\!-w 80 -c 90&amp;check_interval=5&amp;retry_interval=4&amp;max_check_attempts=2&amp;check_period=24x7&amp;contacts=nagiosadmin&amp;notification_interval=5&amp;notification_period=24x7&amp;applyconfig=1&quot;

curl -XPOST "http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&pretty=1" -d "host_name=testapihost&service_description=Mem&check_command=check_xi_service_nsclient\!password\!MEMUSE\!-w 80 -c 90&check_interval=5&retry_interval=4&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"

Add Uptime

curl -XPOST &quot;http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&amp;pretty=1&quot; -d &quot;host_name=testapihost&amp;service_description=Uptime&amp;check_command=check_xi_service_nsclient\!password\!UPTIME&amp;check_interval=5&amp;retry_interval=7&amp;max_check_attempts=2&amp;check_period=24x7&amp;contacts=nagiosadmin&amp;notification_interval=5&amp;notification_period=24x7&amp;applyconfig=1&quot;

curl -XPOST "http://nagios.example.com/nagiosxi/api/v1/config/service?apikey=aw33E58m17bH6d3m4z6Y5V9FNVwJTvDU5tjEAGw7C35iLarr&pretty=1" -d "host_name=testapihost&service_description=Uptime&check_command=check_xi_service_nsclient\!password\!UPTIME&check_interval=5&retry_interval=7&max_check_attempts=2&check_period=24x7&contacts=nagiosadmin&notification_interval=5&notification_period=24x7&applyconfig=1"

Make sure you change the FQDN, API Key, Hostname and password variables for this to work.

I found the configuration wizard very helpful, what I did was add the service I wanted to check, see what the “check command” was and the variables set and then crafted my own curl command to add to Nagios.

Automate SSH Key Rotation on Ubuntu with Ansible

Overview

Test Bed

Requirements

Break Down

ssh_key_rotation.yml

Key Management

Conclusion

Ansible – mySQL root password change on Ubuntu

Test Bed

Requirements

Overview

sql.yml

etc/mysql/.my.cf

global-vars/config.yml

Execution

Other Considerations

Conclusion

Regenerating SID’s using Powershell

The Breakdown

The Script

Considerations

Conclusion

Atlassian Backup to AWS S3

Overview

Requirements

Script

Considerations

Conclusion

Nagios XI – Adding Microsoft Server Services via API

Add Processor

Add Disk

Add Memory

Add Uptime

Archives

Categories

Buy Chris a beer!

February 2019
M	T	W	T	F	S	S
« Nov
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28

Overview

Test Bed

Requirements

Break Down

ssh_key_rotation.yml

Key Management

Conclusion

Test Bed

Requirements

Overview

sql.yml

etc/mysql/.my.cf

global-vars/config.yml

Execution

Other Considerations

Conclusion

The Breakdown

The Script

Considerations

Conclusion

Overview

Requirements

Script

Considerations

Conclusion

Add Processor

Add Disk

Add Memory

Add Uptime

Tags

Archives

Categories

Buy Chris a beer!