DevSecOps Pune Meetup 4

9th March 2019

This is our 4th DevSecOps meetup. More and More and More swag sponsors added to the list. New sponsors: Elastic, Sonatype, Synk along with previous sponsors i.e. Polyverse and Cloudneeti already into the list. The meetup head count was usual and exact. We have been hitting the correct and intended audience.

Qualys Pune was the venue for the meetup. This is the 2nd time we organized the meetup at Qualys. We started at 10.30 am. We had a huge list of topics this time and I was pretty sure not all of them are going to get discussed since the most voted topics were very interesting. The following topics got discussed:

• Machine Learning for Security: This topic got discussed for a good 40 minutes although none of us was an ML engineer and only understood theoretical ML concepts. The topic was chosen more for a discussion as the person suggester of the topic just had the curiosity to see if this can be done. The topic unfolded as people discussed and understood how ML works and how is it being used in the industry. Different tools like TensorFlow, Pandas, etc got discussed. Once we had a clear picture of ML, we moved to understand what security breaches we experienced in our Ops lives in industry. Based on our experience, we framed logic as to how ML algorithm could be written by Data Science engineers by studying security auth logs and application logs. We also discussed how ELK stack could be used to prove a security attack on system and further. We agreed to the part that as an Ops person we can only best provide inputs and prove security issues in systems. An ML engineer should be the best person to provide inputs on what ML algorithms could be used to mitigate security issues.

•  Ansible for DevSecOps CI/CD pipeline: Ansible was suggested multiple times in the past meetups as well but always went not noticed and never got discussed. In this session, we started with Ansible and went on an on for a good 1 hour. We understood how Ansible is best used in different organizations. We also discussed how a bad code in Ansible can mess up and not make any difference between olden days shell script. We discussed Idempotency feature of Ansible. We also discussed as to how Ansible is being used by some firms for Provisioning, Configuration Management and Deployment altogether. We agreed how many used Puppet for multiple jobs and later moving to a different tool was difficult for them. Hence using different tools for specific different jobs could be useful. We discussed a typical CI/CD workflow with Terraform for provisioning, Ansible for CM and Fabric/Capistrano or different language specific deployment tools for easy deployments and rollbacks. We also did a small white board presentation on how CI/CD can be used for all 3 purposes. We also discussed DR strategies, Cloud managed services like auto-scaling services etc.

• Git for securing code : The suggester was a Developer and knew exactly how was not being used in the best manner to ensure security and highlighted some best ways like code review practices, git hooks to be used in Jenkins, linting analysis using pre-commit hooks, static code analysis with SonarQube before merge. We discussed this in details already in CI/CD pipeline above as well.

• Burpsuite :  Burpsuite was a tool that many knew but never got a chance to use as such. The suggester of the topic had extensive experience using it and spoke about multiple possibilities and usecases that could be achieved using Burpsuite. He also highlighted on how precautions need to be  taken before using this tool for testing giving examples of one of his projects. Burpsuite indeed was very interesting for many members and we decided to have a demo session on this.

Takeaways from this session for speakers to prepare were:

•  Session required on ML in security
• Demo on CI/CD in DevSecOps
• Demo on Burpsuite

Many topics that did not get discussed were:

• Nexus scan in Jenkins pipeline
• SonarQube for secure code analysis
• Pentesting with Python
• IOT Security
• Metasploit for pentesting
• Cloud security
• Prowler
• Automobile software release cycle and missing lacking devops chain
• Maven
• Regulations

In the end Shirish wanted to highlight the recent All Intel Chips Open To New 'Spoiler' Non-Spectre Attack

Some clicks :)

Source build Envoy proxy on Ubuntu 18.04

sudo apt-get update
sudo apt-get install openjdk-8-jdk build-essential autoconf libtool cmake ninja-build
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install bazel
wget https://dl.google.com/go/go1.11.5.linux-amd64.tar.gz
tar -xvf go1.11.5.linux-amd64.tar.gz
sudo chown -R root:root ./go
sudo mv go /usr/local
echo "export GOPATH=$HOME/go" >> ~/.profile
echo "export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin" >> ~/.profile
 git clone https://github.com/envoyproxy/envoy.git
cd envoy/
bazel build --package_path %workspace%:/home/<user>/envoy/ //source/exe:envoy-static

RELAX, ITS GOING TO TAKE LONG TIME




To generate the example configurations run the following from the root of the repo:

mkdir -p generated/configs
bazel build //configs:example_configs

RELAX, ITS GOING TO TAKE LONG TIME

I wrote an Ansible playbook too for this. I will publish it later on Github

Easily SCP/Rsync through bastion host or SCP/Rsync through multiple hops

Often we work in environment where we need to copy files or directories from a local system to another server that can be accessed only through a Bastion host. In such cases, typically we transfer from local machine to Bastion and from Bastion to the intended server. This is time consuming, repetitive and unreliable too. There are many ways you can make this automated. I found a way to get this done through SSH tunneling. Here's how it works:

There are 3 machines involved here:

1. localhost
2. Bastion host
3. Intended server

1. Create an SSH tunnel from localhost to the intended host through bastion. The tunnel will be created from port 1234 at localhost. You may choose any other port.
ssh -L 1234:<intended_server>:22 <user>@<bastion-host> cat -
2. In a new tab initiate the file/directory transfer using the tunnel port
scp -P 1234 <file_to_transfer> <user_of_intended_server>@127.0.0.1:~/

As I did this, I realized SCP is very slow in getting the transfer done due to its linear and sequential file transfer behavior. Hence, I used Rsync which made it pretty fast due to its delta based transfer algorithm

rsync -avz -e "ssh -p 1234" <file_to_transfer> <user_of_intended_server>@127.0.0.1:~/

DevSecOps Pune Meetup 3

9th Feb 2019

This is our 3rd DevSecOps meetup. I am glad to see the head count getting even better. Another Goodies sponsor added to the list. Cloudneeti Software sponsored T-shirt and Mugs to best presenter.

For this meetup, we got an exact count again with 2-3 last minute exits. Cloudneeti Software was the venue for this meetup. We started sharp at 10.30. Surprisingly the topics discussed this time were pretty advanced and much looked out for. The following topics got discussed:

•  Kali Linux for security : The topic was super hit and got discussed for pretty long hours. We spoke for a good 20 minutes on Kali Linux and only stopped because the next topics were also equally interesting. We spoke about the old distro Backtrack and how Kali got introduced. Others added their inputs as to how Kali is used by different SecOps people in industry. Overall we all agreed that Kali was a really vast topic to be just discussed and this rather needed a presentation + demo with so many tools within.
• OWASP Top 10 web app security risks: As we discussed Kali, we also spoke about OWASP top 10 in the same discussion and this got prolong for another 20 mins. Not all of us were aware of all OWASP top 10 attacks hence we Googled it just for making notes, however with time limitations we could only discuss the most famous SQL Injection and a little about XSS. We also concluded that this needed a bigger session.
• CI/CD in DevOps Pipeline: DevSecOps CI/CD pipeline was discussed shortly in 1st meetup but we did not deep dive in it then. In this meetup we actually went to the depth of Jenkins, Git, SonarQube, Static code analysis, Container Image security, SecOps role to play, Vulnerable libraries being used by devs and how to resolve this. How Github has inbuilt vulnerability analysis done now and many more details. We also drew the following CI/CD architecture diagram for DevSecOps pipeline.

• Ansible secure key rotation: This was more of a question to the forum as to how this could be done since Chef and Puppet use clients and use their own keys for security, while Ansible uses SSH. We agreed to the fact that the keys must get rotated and many companies do follow this. Ansible's authorized_keys module helps you to rotate keys was the perfect answer suggested.
• Security Patching at scale in cloud: This was again a question and the answer was many companies did this using cloud native tools to create a golden images and other tools to get into the image to verify if the images were CIS compliant this is not just for images but also for Cloud environment and resources in general. Once again Prowler was discussed to evaluate the cloud environment.

Takeaways from this session for speakers to prepare were:

•  Kali Linux hands on demo
• OWASP Top 10 risks
• CI/CD in DevSecOps

A few topics that did not get discussed were:

• Metasploit for Pentesting
• GDPR automation
• Deploying Software securely
• Cloud security trends 2018-19

An important Topic that Budhram did put forth as to what minimum qualifications and expertise do companies look for in a fresher candidate to call him an eligible candidate for DevOps Engineering further. This was a good debate cum discussion that we all spoke about in the end.

This turned out to be a long event in spite of small number of attendees. Ashish, Budhram, Shrikant and Dhiru got goodies to keep discussion more happening and actively participating in all topics as well. Some others got stickers.

Some clicks :)

Copying ssh keys easily

I use VMs/Vagrant a lot in my day work for all sysadmin/devops automation. One of the problems that I always face with the systems is to authorize my server for the 1st time with my master host. If I am using 10 VMs I need to authorize them 10 times? I wrote a small script to automate this process:

1. Create a "list" file and add all IPs and hostnames for the VMs in it.
2. Create a "password" file to write your SSH password in it, you may choose to write the password in the bash, however I feel this gives me the flexibility to add the bash to my source code if need by putting password file in a.gitignore
3. Next create a shell script that will read the IP addresses and the hostnames from the "list" file and password/s from the password file(I generally keep the same password for all VMs for simplicity)
4. Remember 2 commands are useful here ssh-copy-ip and ssh-keyscan. Here's how you use them:

 

ssh-keyscan -H <IP> >> ~/.ssh/known_hosts sshpass -f <password> ssh-copy-id -i ~/.ssh/id_rsa.pub <USER>@<IP>

The ssh-keyscan command command is for gathering the public ssh host key of a VM host specified. After collecting the publich ssh-key it adds it to your localhost. You can verify this by checking the contents of "~/.ssh/known_hosts"

The ssh-copy-id command copies the public key of your default identity (otherwise use -i identity_file for other identities) to the remote host. You can verify this by checking the content on ~/.ssh/authorized_keys in the VM host.

The final script looks like this with a loop:

#!/bin/bash user="vagrant" for ip in `cat ./list`; do ssh-keyscan -H $ip >> ~/.ssh/known_hosts sshpass -f password.txt ssh-copy-id -i ~/.ssh/id_rsa.pub $user@$ip done

That's how my "list" file looks like:

consul-server1 consul-server2 bootstrap-server1 client1 client2 client3 client4 client5 192.168.3.111 192.168.3.112 192.168.3.121 192.168.3.151 192.168.3.152 192.168.3.153 192.168.3.154 192.168.3.155

I purposely add IP as well as hostname as I keep using them interchangeably. I also came to know about ansible authorized_keys module that does the ssh-copy-id task:

- name: Set authorized key for user ubuntu copying it from current user
  authorized_key:
    user: ubuntu
    state: present
    key: "{{ lookup('file', lookup('env','HOME') + '/.ssh/id_rsa.pub') }}"

However, you will still need the the ssh-keyscan here. This script goes handy for ops who keep destroying their local environment and use a new one.This is available on Github: https://github.com/iamrawtion/ansible-autossh

AWS Tagger

Image credits : jdhancock

Tagging in AWS is often not considered useful by many users. Tagging of resources in cloud and DC not only helps us identify resources but it can also do multiple other wonders that one might have never thought about. We don't tag resources in cloud for many reasons, laziness being the topmost reason.

Lets see why tagging is important:

1. Identification and Isolation: Tagging allows identification of resources as to what purpose a specific resource may have been created for. It also allows you to separate resources from each other. e.g. separating different environments.
2.  Automation: When you tag resources with certain values you can ensure that your automation scripts only addresses certain intended resources and not all. e.g execute security patches on certain systems that need to be compliant.
3. Costing: You can identify based on tags as to which resource is costly and also make business decisions based on the results received.
4. Define ownership: You can also understand based on proper tags as to who are the stakeholders for a certain resource or group of resources.
5. Versioning: Sometimes when you need certain resources to be preserved based on its state, you may also versionize them based on tagging. Although AWS provides versioning mechanism for a few services, it may not be applicable to all of them.

 In many organizations although the importance of tagging is understood a lot later. Until then its too late to start tagging and it becomes almost always a manual process to tag all the resources. Or you may need to write complex programs to identify systems and tag them as per your requirement. Thankfully, AWS Tagger comes to rescue if you have a requirement to tag your AWS resources. You may also bulk tag them to avoid a lot of manual work. So how do we do this.

Its a 3 step process to Bulk tag resources:

1. Collection : This is a simple process. Here all you need to do is, collect all the resources in a file. Hereafter you may process this data. AWS Tagger heavily depends of resource ID's of all the resources you create. Resource ID's are further used to implement all the tags. To get the resource ID's for all the resources, simply login to your AWS account and navigate to https://resources.console.aws.amazon.com/r/tags . On this page, you are given a field to enter the region for the resource you want to choose and all choose the types of resource. Choose "All resource types" here and click on "Find Resources" button. Click on the "Download" button to download the CSV data generated.
2. Identification and filtering: I recommend this step particularly to filter the data so that AWS Tagger can act on individual resources. Here you may use your excel skills to separate data based on resource types.
3. Tagging: Once the resources are separated, you may start executing AWS Tagger scripts as per the documentation provided on their Github page.

DevSecOps Pune Meetup 2

For the 2nd DevSecOps meetup, we already had our first swag sponsor. For DevOps Pune, I received swags from Docker and Ansible. Hashicorp was also planning to send a few.

With DevSecOps our 1st swag sponsor was Polyverse :) I couldn't resist posting these.

For this meetup, I changed the RSVP format to get an exact count. I was expecting to start soon at a bigger location. We couldn't risk wasting of resources. Lean coffee needs logistics to take care of and hence needs to be addressed really well. Everyone who RSVP' at the meetup page was informed to Call/SMS/WhatsApp the organizers to book a slot. So the Meetup page RSVP meant nothing.

Qualys Pune, was our venue, logistics and refreshment sponsor this time. When I 1st told them about the meetup, this is how the they arranged the seating :) :

I told them about the format and later we changed the seating to best suit the format. 

We had 12 RSVP and 10 attendees this time with just 1 last minute informed cancellation. That was a perfect number . We also made sure that the refreshment we take was a packed and long lasting one so we don't waste food. 

With 10 attendees we stared pretty much on time and this time we had a huge list to discuss and the participants were from mixed domains unlike last meetup. This time we had QAs, Support Engineers, DevOps, Consultants and Developers altogether.

The following topics got discussed:

• Understanding PKI - Public Key Infra (How SSL Works?)
• CIS Benchmarking
• SOAR (Security, Orchestration, Automation and Response)
• AWS Compliance
• Securing serverless in Azure (Function as a Service)
• Debian Linux and Contributing to it
• Microservices with an example

SSL and PKI got discussed a lot since Muneeb Shaikh really explained concepts that we were unaware of that goes behind PKI formation and how public and private keys work. A 5 minutes discussion prolonged to a good long 30 minutes with inputs from everyone.

SOAR was a new process and concept that Rahul spoke about and worth reading for everyone. CIS compliance was a hit one this time too. We understood that it was topic worth presentation hence added to the DevOps Pune meetups Agenda. Later we spoke about AWS compliance and how Prowler could be used to ensure compliance in AWS. We also discussed importance of good naming conventions and Tagging in AWS. There was a chaos when we spoke about both serverless and microservices resulting to a debate topic that we discussed at the end of the meetup. The final closure was on Debian systems and how we would contribute to it.

Takeaways from this session for speakers to prepare were:

•  PKI and SSL encryption
• Microservices

A few topics that did not get discussed were:

• Types of Security and importance of each
• GRC - Governance, Risk and Compliance
• Security Testing with Selenium
• Achieving CI/CD with Ansible

This turned out to be a long event in spite of small number of attendees. Muneeb got Polyverse T-Shirt to keep PKI discussion happening and also actively participating in all other topics as well. Some others got stickers.

Some clicks :)

DevSecOps Pune Meetup 1

After some good success, a huge gap and some mixed learning experiences with organizing DevOps Pune meetups, I decided to start with DevSecOps Pune meetup. This was mainly since I was exploring possibilities in Information Security world. The idea started after seeing DevSecOps Seattle Meetup and the learning experience I had their simply by reading their updates. I saw regular posts on Facebook about this meetup group from my long time mentor Archis Gore. I was still confused whether to start a meetup in Pune or just stick to Seattle meetup and attend it virtually. You don't always have to be an organizer to learn.

However, Archis told me that the Seattle meetup could not be attended virtually, as the format could not support virtual attendance. This was a Lean Coffee format. Something different for me. On learning more about the format, it sounded really like a plan to start a similar meetup in Pune. Archis was here in Pune in October 2018, when I met him to understand the organizer's roles in this format. And then, and we were in or a great start. I got Rahul Khengare with me this time as a Co-organizer and started the meetup group. Cloudneeti helped us sponsor the meetup group.

The first meetup I knew would be a small one with limited attendees. I expected less than 10 attendees to show up and the RSVP count always go wrong. I remember wasting lot of food and other resources in the past due to incorrect RSVP count. I chose a location to the central Pune so that its easy to commute to everyone. Thanks to Bobby Jadhav for sponsoring the venue, i.e HauteBook's office.

We had 5 attendees in this meetup as expected. The count was not important. What was more important was whether good topics came up. With Lean Coffee we expect every participant to come up with good topics to vote for and speak about. The topics that got discussed were:

• Docker image security and its challenges (Highest voted)
• DevSecOps CI/CD pipeline with Kubernetes
• Cloud Security
• Metasploit - Kali Linux

We discussed a lot about Docker security and Snyk, Twistlock being the docker security tools were also explored further. How CIS compliance helps cloud security and the recent how engineers disable SELinux first on any system and that is a bad practice. Although just 5, the discussion went along for a huge 2 hours and it was indeed a wonderful learning experience.

We also decided later that the takeaways from these lean-coffee format will end up becoming speaker-attendee format topic for us to deep dive into important topics further. This was a great takeaway since with DevOps meetups we only spoke about what the speaker was best at, and may or may not be community learning requirement. Takeaways from this session for Speakers to prepare about was:

• CIS Compliance
• SELinux / Apparmour

These topics were added as Topics on DevOps Pune meetup and the hunt for the speaker started there. A few topics that did not get discussed due to lack of time were:

• iptables
• Ansible, Terraform and CI/CD for pod deployments on AWS
• AWS security alternatives
• Security compliance
• Securing Nginx

Overall it was a wonderful learning experience. Cheers to all the attendees.

Some clicks :)

Tools mentioned in Devops Weekly

All credits to http://www.devopsweekly.com/ looked after by Looked after by Gareth Rushgrove.



Kashti is a dashboard for anyone using Brigade, a tool I’ve mentioned previously for building elaborate pipelines composed of multiple containers running on Kubernetes.

https://open.microsoft.com/2017/12/06/kashti-kubernetes-open-source-microsoft/
https://github.com/Azure/kashti


Jo is a handy utility which can output JSON based on arguments to the CLI tool. Handy for API testing and other use cases.

https://github.com/jpmens/jo


Kitchen Terraform is a plugin for Test Kitchen to allow for testing resources provisioned by Terraform.

https://newcontext-oss.github.io/kitchen-terraform/


Escape is a toolset for release engineering, life-cycle management and continuous delivery of software platforms and artefacts. Unlike some similar tools it hands off much of the work to other lower-level platforms or command line tools and focuses on the integration piece.

https://github.com/ankyra/escape/
https://escape.ankyra.io/docs/what-is-escape/


Metaparticle is a new project aiming to be the standard library for distributed systems. It takes advantage of the capabilities of Kubernetes to expose features like master election and container packaging and scale out, with current implementations in .NET, Java and Javascript.

https://metaparticle.io/posts/welcome-to-metaparticle/
https://metaparticle.io/


SockGuard is a eat new project which provides a proxy to the Docker socket which exposes some useful access control features.

https://github.com/buildkite/sockguard

 MetalLB is a Kubernetes-native load balancer for environments where the cloud-provider load balancer integration isn’t suitable, for instance when running a bare-metal cluster.

https://github.com/google/metallb


SPIFFE is a set of open-source standards for securely identifying software systems in dynamic and heterogeneous production environments. It’s still in development, but a useful shared solution to a common problem.

https://spiffe.io/
https://github.com/spiffe/spiffe


The Chaos Toolkit is a set of tools for describing and running chaos engineering style experiments against your systems. The tutorial does a good job of introducing the toolset.

http://chaostoolkit.org/
https://github.com/chaostoolkit/

 container-diff is a new tool for comparing container images, either locally or from a remote repository. It detects packages from a number of package management tools, as well as filesystem changes. Useful for debugging image build problems.

https://opensource.googleblog.com/2017/11/container-diff-for-comparing-container-images.html
https://github.com/GoogleCloudPlatform/container-diff


Open Policy Agent is an interesting new project providing which provides a DSL for describing policy, and software to ensure described systems remain compliant. Examples include managing Kubernetes API admission, Terraform permissions and SSH authorization.

http://www.openpolicyagent.org/

I’ve seen (and build) a few integration testing setups for container images, but DockerSpec aims to come with everything out-of-the-box. RSpec helpers for testing Dockerfiles, image builds, running containers, Compose and more.

https://github.com/zuazo/dockerspec

Neugram is a new scripting language, based heavily on Go, but with some useful tricks to cut down on some of the syntax. It also provides a REPL and shebang support.

https://neugram.io/blog/neugram-briefly
https://neugram.io/


Psykube provides a slightly higher-level abstraction that the raw APIs for describing applications running on Kubernetes.

https://github.com/CommercialTribe/psykube


CIM looks like a handy tool for anyone managing AWS CloudFormation templates. Rather than try and provide an abstraction, CIM provides a number of helpful user interface tools which make working with templates easier.

https://medium.com/@rgfindley/meet-cim-cloud-infrastructure-manager-bc8bcfe0593c
https://github.com/thestackshack/cim

lstags is a handy tool for comparing Docker images between repositories, as well as automatically syncing images between repositories.

https://github.com/ivanilves/lstags


Brigade is a new workflow tool for Kubernetes environments, allowing for writing JavaScript to define pipelines for running containers. Supports timers, listening to events from services like GitHub or Docker image repositories and more.

http://brigade.sh/
https://github.com/azure/brigade/

OpenMetrics is a public working group looking to determine a standard for exposing metrics data, influenced by the Prometheus exposition format. I’m a big fan of adhoc agreement so hopeful this takes off.

https://github.com/RichiH/OpenMetrics

Kapitan is another Jsonnet based configuration tool for Kubernetes. As well as configuration templates it provides built-in support for generating documentation and scripts, and a tool for separating data from configuration.

https://github.com/deepmind/kapitan


Tarmak is a new toolkit for managing and provisioning a Kubernetes cluster. It makes use of Packer, Puppet, Terraform and Docker to provide a degree of platform independence, but provides a higher-level CLI interface and good defaults for a secure, CI/CD ready cluster.

http://docs.tarmak.io/en/latest/
https://github.com/jetstack/tarmak

Kedge aims to provide a high-level data-centric interface for describing Kubernetes applications, with a focus on good defaults and familiarity for folks already using the lower-level APIs.

http://kedgeproject.org/
https://github.com/kedgeproject/kedge


The Docker Version Manager (dvm) is a cross-platform command-line tool that helps you install and switch between Docker clients. Handy if you’re working with a range of different server versions.

https://howtowhale.github.io/dvm/
 

Kubesh is a handy tool for anyone managing several Kubernetes clusters. It allows for spawning new shells with different Kubernetes contexts.

https://medium.com/@lestrrat/handling-multiple-kubernetes-clusters-using-kubesh-a7e2a1606bfa
https://github.com/lestrrat/kubesh


Vespa is a new open source engine for executing and serving computations over large data sets in real time. It allows you to write and persist any amount of data, and execute high volumes of queries over the data which typically complete in tens of milliseconds.

http://vespa.ai/

Chart Museum is an open source Helm Chart repository, for anyone using the helm package manager with Kubernetes. It supports S3, Google Cloud Storage backends and local storage, and the README contains details for how to get up and running.

https://github.com/chartmuseum/chartmuseum


Ducktape is a framework for writing high-level system integration tests for distributed systems, originally written to test Kafka.

https://github.com/confluentinc/ducktape
https://ducktape-docs.readthedocs.io/en/latest/


Jaeger is an open source system for monitoring distributed systems, using the OpenTracing API.

https://github.com/jaegertracing/jaeger

Pumba is a tool for chaos testing in a Docker environment. Supports interfering with network traffic to simulate connectivity issues and stopping containers.

http://blog.terranillius.com/post/pumba_docker_chaos_testing/
https://github.com/gaia-adm/pumba


AWX provides a browser-based user interface and API for Ansible. It represents part of the Ansible Tower product, now available as an open source project.

https://github.com/ansible/awx


Webhooks can be a useful way of integrating different software, but how best to run a set of hooks? Enter webhook. Describe hooks in a simple configuration file and point it ad executables and it exposes a simple HTTP/S interface to run them.

https://github.com/adnanh/webhook

OK Log is a distributed and coördination-free log management system for large clusters. It's designed to be a building block: easy to understand, easy to operate, and easy to extend.

https://github.com/oklog/oklog


Cruise-control (not the CI system) is a new tool to fully automate the dynamic workload rebalancing and self-healing of a large kafka cluster.

https://engineering.linkedin.com/blog/2017/08/open-sourcing-kafka-cruise-control
https://github.com/linkedin/cruise-control


Gosu solves a specific problem for anyone building Docker images, namely running an application as a different user but without interfering with signal processing and TTY.

https://github.com/tianon/gosu


PlantUML is a text-based diagramming tool, and this project provides primitives for diagraming with AWS service. Lots of good examples in the README.

https://github.com/milo-minderbinder/AWS-PlantUML

 Squash looks like an interesting idea to build a debugger for microservices. Currently it supports Kubernetes and VSCode. The rolling demo gives you an idea of how it works.

https://github.com/solo-io/squash


Serverless applications have typically relied on platform-specific services. For those interested in more cross-platform approaches the new event gateway application from the folks behind the serverless framework looks useful.

https://serverless.com/blog/introducing-serverless-event-gateway/
https://github.com/serverless/event-gateway


Kubernetes Deploy is a new, high level, tool for deploying applications to Kubernetes. It focuses not just on submitting changes, but helping the user understand exactly what changed.

https://github.com/Shopify/kubernetes-deploy

Functions as a Service (or FaaS) is a new serverless framework aimed at running on top of Docker or Kubernetes.

https://blog.alexellis.io/introducing-functions-as-a-service/
https://github.com/alexellis/faas


OpenEBS provides containerized block storage with per-container (or pod) QoS SLAs, tiering and replica policies across AZs and environments, and predictable and scalable performance.

https://github.com/openebs/openebs
 

Supercronic nicely solves the problem of running cron jobs in containers, providing a cron implementation which happily takes environment variables and outputs straight to stdout and stderr amongst other features.

https://github.com/aptible/supercronic


Atlantis is a workflow tool for teams working with Terraform. It provides some nice features to prevent common mult-users issues and integrates with GitHub pull requests.

https://github.com/hootsuite/atlantis

Manifesto is a new tool for storing and retrieving Docker image metadata alongside the image in question in a repository, including on Docker Hub.

https://github.com/aquasecurity/manifesto


Terraboard is a handy dashboard for anyone using Terraform. It allows for visualising and querying terraform state from a simple browser-based interface.

https://github.com/camptocamp/terraboard

Kedge is looking to provide a concise application definition for kubernetes. Higher-level than the full API wireformat, Kedge describes a YAML file format which can be expanded by the bundled command line kedge tool.

http://kedgeproject.org/
https://github.com/kedgeproject/kedge


Jsonnet is cropping up in a number of places for managing configuration so it’s nice to see a native unit testing framework.

https://github.com/yugui/jsonnetunit


Buildah is a new tool for building OCI-compatible and Docker-compatible images. It provides a series of commands for building images up, and can also use existing Dockerfiles.

https://github.com/projectatomic/buildah


Karn is a tool for creating seccomp and apparmor profiles. It provides a simple TOML based description language and a tool to convert multiple declarations to the more complex profiles.

https://github.com/GrantSeltzer/karn

K8Guard is an auditing system for Kubernetes clusters. It monitors different entities, like image size, third-party repository usage, missing metadata, etc. on your cluster for possible violations.

https://target.github.io/infrastructure/k8guard-the-guardian-angel-for-kuberentes
https://k8guard.github.io/


Kubicorn is another project in the Kubernetes management space. A library more than a tool to use directly, kubicorn is a new projects but with some interesting design goals.

https://github.com/kris-nova/kubicorn

Smith is a command line utility for building microcontainers from rpm packages or oci images. The resulting image only contains the process to be run and its direct dependencies, is particular careful about file ownership and permissions and can be run read-only.

https://github.com/oracle/smith


  Kubeval is something I’ve been hacking on recently to validate Kubernetes config files. Especially if you’re generating those files with templates, or supporting multiple versions of Kubernetes, this might be handy.

https://github.com/garethr/kubeval


Kube Shell is exactly what you’d expect, a shell for interacting with Kubernetes. Auto completion, command suggestion, history, inline documentation and more.

https://github.com/cloudnativelabs/kube-shell

Kube-bench is a handy tool for checking a Kubernetes installation against the published security guidelines in the CIS Kubernetes 1.6 Benchmark.

https://github.com/aquasecurity/kube-bench


ImageWolf is a new tool to help quickly distribute Docker images around a cluster using BitTorrent. ImageWolf runs on Docker Swarm and can integrate with both the Docker Hub and local registries.

http://container-solutions.com/lightning-image-deployment-imagewolf/
https://github.com/ContainerSolutions/ImageWolf


Buildkit is a new, low level, tool for building container images. The intention is to provide the backend for build in a separate package, allowing for compatible but new and different build interfaces to be built.

https://github.com/moby/buildkit


Managing secrets in a way that feels native to Kubernetes has been much discussed. Sealed Secrets therefore looks like an potential approach with some interesting properties.

https://github.com/bitnami/sealed-secrets
 

A nifty tool for anyone integrating Kubernetes into an existing environment, pam_hook allows for using unix-style users and groups and LDAP to provide tokens for use with the Kubernetes API.

https://github.com/bjhaid/pam_hook


CredStash is a very simple, easy to use credential management and distribution system that uses AWS Key Management Service (KMS) for key wrapping and master-key storage, and DynamoDB for credential storage and sharing.

https://github.com/fugue/credstash


Have you ever wanted to search through a filesystem but forgotten the magic incantations to find or ack? Fsql provides an interesting SQL alternative for a wide range of file system queries.

https://github.com/kshvmdn/fsql


Welder is a simple script execution tool, intended for bootstrapping server instances with as few features as possible.

https://github.com/pch/welder

Draft is a new high-level tool for managing kubernetes applications, targeting the local development experience with packs for common development environments.

https://github.com/Azure/draft


An interesting approach to creating distroless container images, using Bazel to build a minimal base image and accompanying language runtime images for Java, Python and more.

https://github.com/GoogleCloudPlatform/distroless


K2 is a new tool for deploying a Kubernetes cluster on top of CoreOS using Terraform and Ansible under the hood.

https://github.com/samsung-cnct/k2
https://github.com/samsung-cnct/k2cli
 

Istio is a new service mesh for microservice applications based on Kubernetes. It provides a standard way to connect, secure, manage and monitor microservices, based on the Envoy proxy.

https://istio.io/blog/istio-service-mesh-for-microservices.html
https://github.com/istio/istio



Lumogon is a little something I’ve been working on at Puppet recently. It’s an open source tool for inspecting running Docker containers. Initially it can report on software packages installed in the container, but we have lots more ideas for other capabilities.

https://github.com/puppetlabs/lumogon
https://lumogon.com/


Gixy is a static analyzer for Nginx configuration, intended to detect common security misconfigurations. This would be great in a CI pipeline.

https://github.com/yandex/gixy

If you’re using lots of Kubernetes you might have different contexts you want to manage. Enter kubectx, a handy tool for flipping backwards and forward, with support for aliases and shell completion.

https://ahmet.im/blog/kubectx/


BeePing is a service to allow for HTTP Monitoring via API. When running BeePing you can hit it’s API and it will confirm various properties like performance or SSL cert expiry for the requested domain.

https://github.com/yanc0/beeping


JSONNET is a handy programming language for creating JSON, useful for all sorts of configuration tasks. Kube.libjsonnet is a library for helping manage Kubernetes configs in a sane manner.

https://github.com/heptio/kube.libsonnet

Gordon is a tool for deploying and managing AWS Lambda functions, backed by CloudFormation. It provides a nice high-level abstraction over the many underlying services, and the documentation has lots of useful examples.

https://github.com/jorgebastida/gordon

Kubemr is a mapreduce framework for data processing on Kubernetes. It’s an interesting example of using the various Kubernetes primitives to build higher-level systems.

http://www.sajalkayan.com/post/kubemr.html
https://github.com/turbobytes/kubemr


Vaul UI is a user interface for the Vault secrets management application. It’s neatly packaged as either a local application or a shared web app, and supports a number of backends for authentication.

https://github.com/djenriquez/vault-ui


Deputize neatly integrates PagerDuty and LDAP, by exporting on-call information from a PagerDuty schedule to a configurable LDAP group.

https://blog.threatstack.com/balancing-security-and-your-on-call-rotation-using-deputize
https://github.com/threatstack/deputize


Weathervane is a new application for conducting a performance benchmark against a virtualized or cloud environment.

https://blogs.vmware.com/performance/2017/04/weathervane-performance-benchmarking-now-open-source.html
https://github.com/vmware/weathervane
 

Cerebro is an alerting system build to integrate with Graphite. The service layer currently integrates with Seyren to store alarms, schedule checks and send alerts

https://github.com/voyages-sncf-technologies/cerebro


Codeflow is a new deployment pipeline tool aimed at 12-factor apps on Kubernetes. It features a plugin based system for extending the core functionality too.

https://codeflow.checkr.com/
https://github.com/checkr/codeflow


Kube Applier is another kubernetes deployment tool, this time aimed at running kubernetes configuration files on a schedule and reporting on any drift.

https://blog.box.com/blog/introducing-kube-applier-declarative-configuration-for-kubernetes/
https://github.com/box/kube-applier

springboard is a cli utility to help get your secrets into vault. It allows for storing encrypted files in source control, and publishing those secrets securely in Vault.

https://github.com/benschw/springboard


One of the issues for some with the unikernel approach is the need to use specific programming languages. This demonstration shows a possible alternative, in this case transpiling PHP.

https://github.com/tfjmp/php2uni
https://www.cl.cam.ac.uk/research/srg/opera/publications/papers/2017ic2ePHP2Uni.pdf
 

Dockerscan is another security scanning tool for use with Docker images, but one focused on gathering information useful for an attacker. Useful for defence in better understanding your exposure.

https://github.com/cr0hn/dockerscan


Related to the opentracing presentation above, Loki is a new opentracing compatible tracing application with similar design sensibilities to the prometheus monitoring system.

https://github.com/weaveworks-experiments/loki


DNSControl is a system for maintaining DNS zones for different provides (including Route53, CloudFlare, and Gandi), using a custom DSL and runtime. It apparently generates the most beautiful BIND zone files ever.

https://github.com/StackExchange/dnscontrol
https://stackexchange.github.io/dnscontrol/


LXDock is a tool intended for building a local development environment. It acts as a thin wrapper around LXD and provides a YAML based users interface for describing what you need.

https://github.com/lxdock/lxdock
https://lxdock.readthedocs.io/en/stable/


If you’re writing tools against the AWS APIs testing can be a challenge. Localstack provides an easy-to-use test and mocking framework with local copies of several AWS APIs including Lambda, DynamoDB, S3, API Gateway and more, all packaged as a Docker image.

https://github.com/atlassian/localstack


FireHOL is a language (and a program to run it) which builds secure, stateful firewalls from easy to understand, human-readable configurations. It also includes FireQOS which provides extensions for traffic shaping too.

http://firehol.org/


Ctop is a top-like interface for container metrics, connecting to a Docker socket and presenting information about container memory, CPU and network usage.

https://bcicen.github.io/ctop/
https://github.com/bcicen/ctop


Kubecfg is a tool for managing complex Kubernetes configurations, by providing a nice wrapper around jsonnet templates.

https://github.com/anguslees/kubecfg



Cloud Custodian is a tool to help with keeping an AWS account to a set of defined policies, like ensuring tags are applied or that ec2 instances have encrypted volumes.

https://github.com/capitalone/cloud-custodian


Prophet is a tool for producing high quality forecasts from time series data. It is focused on allowing for accurate forecasting to be done by non-experts

https://research.fb.com/prophet-forecasting-at-scale/
https://github.com/facebookincubator/prophet


Trilogy is a new tool for testing database logic. Test cases are represented by markdown files and it’s designed to be used with a continuous integration system.

http://engineering.pivotal.io/post/trilogy-the-sql-testing-framework/
https://github.com/pivotalsharedireland/trilogy

An interesting set of tools for managing security in Google Cloud, focused on enforcing firewall rules across multiple projects.

https://labs.spotify.com/2017/02/22/google-cloud-security-toolbox/
https://github.com/spotify/gcp-firewall-enforcer
https://github.com/spotify/gcp-audit

awless is a CLI for Amazon Web Services which aims to be fast, powerful and easy-to-use. Useful to exploring an existing infrastructure as well as ad-hoc commands.

https://github.com/wallix/awless


Mach is a new build tool trying to take some of the things about Make that make it great and bring to them projects (including Java and CLojure projects) that use a hierarchical file system.

https://github.com/juxt/mach
 

HubCommander is a new chatbox for GitHub organization management. You can create new repos, grant permissions, enable Travis all from your chat room.

https://github.com/Netflix/hubcommander
http://techblog.netflix.com/2017/02/introducing-hubcommander.html

Minishift is a quick way of getting a local OpenShift cluster up and running in a VM with the minimum of fuss.

https://github.com/minishift/minishift


LogHub is a new log forwarder with some powerful data pipeline features. It received events from external sources, process them and send them, and is configured with it’s own DSL.

https://github.com/fbacchella/LogHub


zetcd is a simple forwarder for etcd which allows you to use etcd as the backend for zookeeper clients.

https://github.com/coreos/zetcd


Beringei is a high performance, in-memory storage engine for time series data.  It’s been designed specifically for storing health and performance monitoring data with very high write loads.

https://code.facebook.com/posts/952820474848503/beringei-a-high-performance-time-series-storage-engine/
https://github.com/facebookincubator/beringei

Vuls looks like an excellent, modern, vulnerability scanner for Linux and FreeBSD. Run either interactively or on cron it has support for scanning containers as well as the host OS.

https://github.com/future-architect/vuls


Tig is a text mode interface for Git. Something more than the command line can be useful when exploring a repository or learning git, but not everyone wants a full-blown GUI.

https://jonas.github.io/tig/


A build toolchain for MirageOS unikernels built around Docker. Integrates Solo5 to provide targets for kvm, ukvm and qemu.

https://github.com/mato/docker-unikernel-runner


Pipenv is a new tool for python dependency management. It’s aim is to combine pip, the new Pipfile, virtualenv and ideas from other language systems into a single polished user interface.

https://www.kennethreitz.org/essays/announcing-pipenv

Cernan is a new telemetry and logging aggregation server. It exposes multiple interfaces for ingestion (eg. statsd, graphite, line-oriented log files ) and can emit to multiple aggregation sources while doing in-flight manipulation of data.

https://medium.com/@bltroutwine/announcing-cernan-28c245a12f91#.4nbmtqh5q
https://github.com/postmates/cernan/


Hellogopher is a Makefile that makes your conventional Go project build from anywhere, for anyone, with just make. Aimed more at the occasional Go programmer, it makes the point that Go developers should know and use GOPATH but it shouldn't be the first thing they are exposed to.

https://github.com/cloudflare/hellogopher

Kubeplay is a REPL for the Kubernetes API. It allows for exploring and displaying information about Kubernetes resources using a Ruby DSL, with some commands for modifying resources too.

https://github.com/errordeveloper/kubeplay


Kubernetes provides a number of ways for third parties to extend it, kubevirt runs with that and is adding the ability for Kubernetes to manage virtual machines, adding the new VM resource amongst others.

https://github.com/kubevirt/kubevirt


Systemdlogger is a small tool to used to export systemd journald logs to an external service, for example cloudwatch or elasticsearch. It’s designed to be simple, simpy running out of cron.

https://github.com/techjacker/systemdlogger


Screwdriver is a new Continuous Delivery build system. It runs atop Kubernetes or Docker Swarm and provides a web UI as well as the ability to define pipelines in a YAML dataformat.

https://yahooeng.tumblr.com/post/155765242061/open-sourcing-screwdriver-yahoos-continuous
http://screwdriver.cd/

 

Cherami is a new distributed, scalable, durable, and highly available message queue system. The accompanying blog post does a good job of explaining why another queuing server, and the animations give you a good sense of the design.

https://eng.uber.com/cherami/
https://github.com/uber/cherami-server


Grumpy is a Python to Go source code transcompiler and runtime. That means you can convert your Python code to a Go static binary, avoiding the python global interpreter lock and runtime dependency. Still early days but very interesting.

https://github.com/google/grumpy

Conductor is a new microservices orchestration engine. It provides a blueprint DSL to describe a process flow, and then wires up individual microservices as well as providing a useful visualisation of the topology.

https://github.com/Netflix/conductor
https://netflix.github.io/conductor/


Burry is a backup and recovery tool for etcd and zookeeper. As these tools are often used as critical components of some systems being able to use backups to recover from some failure modes is likely to be critical.

https://github.com/mhausenblas/burry.sh


Alerta is a powerful addition to a monitoring setup, providing a single place to manage alerts. It integrates with a wide range of monitoring systems and does simple de-duplication and correlation.

http://alerta.io/


Banshee is a real-time anomaly detection system for periodic metrics. It integrates nicely with statsd and can issue alerts as well as providing a nice dashboard.

https://github.com/eleme/banshee


Ostent collects metrics (memory, CPU, network IO, disk space, etc.) and forwards them to tools like InfluxDB, Graphite or Librato. It also features a simple user interface.

https://github.com/ostrost/ostent
 

A lot of information is stored in a github repository as it’s used, and the git CLI tools provide one interface to that information. GitQL provides a more SQL like read-only view, handy for repository anaylsis.

https://github.com/cloudson/gitql



I’m seeing lots of interest in using Kubernetes as a platform on which to run serverless architectures. Fission looks like a good early approach to this, although it’s described as being in “early alpha”.

http://fission.io/
https://github.com/platform9/fission


pREST is a handy looking tool for very simply exposing a PostgreSQL database via an HTTP interface.

https://github.com/nuveo/prest


Anistrano is a port of the features and usage patterns of Capistrano 2.0 to Ansible, providing a deploy and rollback role to aid with application deployment.

https://github.com/ansistrano/deploy


Jid is a simple tool for exploring JSON documents interactively. Very handy if you’re dealing with large JSON documents, say with the AWS CLI responses.

https://github.com/simeji/jid


Grafanalib is a nifty library for generating Grafana dashboards from simple Python scripts.

https://github.com/weaveworks/grafanalib


runv is analogous to the runc container runtime, except runv runs your container on top of a real hypervisor. Currently supports kvm, xen and virtualbox. Useful if you need additional isolation guarantees.

https://github.com/hyperhq/runv


Funker is an interesting experimental approach to packaging up pieces of your application (functions or otherwise) as Docker containers and have them run on-demand on a swarm.

https://github.com/bfirsh/funker


ProxySQL is a high performance MySQL proxy which allows for query caching, routing queries to different clusters, smart query firewall rules and more.

https://github.com/sysown/proxysql
http://www.proxysql.com/

Storing credentials in your shell profile and exporting them as environment variables is incredibly handy but very insecure. Envchain looks to provide the same ease of use but to use  OS X Keychain or gnome-keyring as the backing service.

https://github.com/sorah/envchain


Pinpoint is an open source Application Performance Management tool for large-scale distributed systems. It features some tracing capabilities, the various screenshots give a good sense of its capabilities.

https://github.com/naver/pinpoint


IronFunctions is an open source serverless platform which supports running Docker images via a simple API. It’s quite early but the documentation walks you through how to try it out quickly.

https://github.com/iron-io/functions
 

If you’re building your own dashboards it can be easier to start with some sort of framework. Enter this project, which provides a set of responsive templates specifically aimed at making dashboards quickly.

https://github.com/keen/dashboards


RackHD describes itself as a stack for enabling automated hardware management and orchestration. It’s an opinionated set of existing tools, bundled together with a high-level API server.

http://rackhd.readthedocs.io/en/latest/



I’ve talked before about the potential for alternative container build tools. One such recent addition is box, which builds images using a subset of Ruby, using mruby embedded in a Go application.

https://github.com/erikh/box


And for those more familiar with Puppet the recent Puppet image_build module allows for building containers images direct from Puppet’s domain specific language.

https://github.com/puppetlabs/puppetlabs-image_build


A handy tool announced at PuppetConf, octocatalog-diff allows for some interesting testing of Puppet code, in particular helping to reduce the risk of refactoring existing code.

http://githubengineering.com/octocatalog-diff-github-s-puppet-development-and-testing-tool/
https://github.com/github/octocatalog-diff


Lots of people will tell you containers are made of cgroups and namespaces, but how do you dig into those? Enter cinf which provides a handy tool for low-level container prodding.

https://github.com/mhausenblas/cinf

Gradle is a popular and powerful build tool used in a number of different language environments, and PyGrade brings that capability to Python.

https://github.com/linkedin/pygradle


Gogeta is a reverse proxy backed by etcd. It provides real time dynamic reconfiguration of routes without having to restart the process.

https://github.com/arkenio/gogeta

For lots of workflows setting the GitHub master branch to protected is a sensible choice, but it’s relatively recent addition and you may have lots of repos. Enter Petter, a simple tool to change the setting for all repos in a given organisation.

https://github.com/jessfraz/pepper


Zentral is a nice looking tool for host inventory and acting on inventory data, built atop osquery, Santa, Sal, ElasticSearch and with integration with various other tools.

https://github.com/zentralopensource/zentral/wiki


Cillium is another container networking approach, but one which uses experimental kernel features like BFP and XDP. It aims to provide fast in-kernel networking along with string security policy enforcement.

https://github.com/cilium/cilium
 

Doorman is a very interesting approach to rate limiting for internal services, pushing some of the authority to the client but providing a central service knowledgable about the needs and wants to the different clients.

https://github.com/youtube/doorman


Babushka is a tool for automating systems administration tasks which makes explicit the configuration change and the test to verify the change.

http://babushka.me/


LambCI is a CI and build system designed to run on AWS Lambda. This makes for some interesting properties, including potentially large cost savings depending on your build profile.

https://medium.com/@hichaelmart/lambci-4c3e29d6599b#.fhhti6fbs
https://github.com/lambci/lambci


Static site generators are nothing new, but Trestus is interesting as it’s very domain specific. It aims to build status pages, for communicating with users around outages, with the data coming from Trello. If you’re using Trello internally to manage an outage that could be a nice workflow.

https://canonical-ols.github.io/trestus/


SQLint is a simple SQL syntax checker, ideal for sanity checking SQL files as part of a continuous integration workflow.

https://github.com/purcell/sqlint

Envoy is a new proxy and network bus that runs on every host and abstracts the network by providing common features (load balancing, circuit breaking, service discovery, etc.) for your applications and services.

https://eng.lyft.com/announcing-envoy-c-l7-proxy-and-communication-bus-92520b6c8191#.vlashkab0
https://lyft.github.io/envoy/


Minio is an object storage server which provides compatibility with Amazon S3. It provides a nice command line and web based interface and can serve an existing directory of files.

https://github.com/minio/minio


CoreDNS is a new DNS based service discovery tool, intended to be the next generation of the popular SkyDNS project. It’s built using Caddy and already comes with several optional middleware components.

https://coredns.io/
 

I love Make and you should too. But writing maintainable Make files is often a challenge. Enter Checkmake, an opinionated linting tool for Make

https://github.com/mrtazz/checkmake


My Looking Glass is an open source software utility which combines the functions of several different network probes (HTTP, ping, DNS, port scanning and more) in one network diagnostic tool.

http://mylg.io/


Pshtt is a handy tool for anyone managing a large number of sites and trying to enforce HTTPS best practices like use of HSTS or specific cypher suites.

https://github.com/dhs-ncats/pshtt

Terraform is an increasingly common tool for describing cloud infrastructure in code, but using it in teams requires some sharing of state. Enter Terragrunt, an open source service you can run to support locking and sharing state between users.

https://blog.gruntwork.io/add-automatic-remote-state-locking-and-configuration-to-terraform-with-terragrunt-656a57565a4d#.ejrf45azt
https://github.com/gruntwork-io/terragrunt


An interesting approach to creating an Apt package repository on S3, using Lambda to build the package index automatically when you upload new packages.

http://webscale.plumbing/managing-apt-repos-in-s3-using-lambda
https://github.com/szinck/s3apt


Skopeo is a command line utility for various operations on container images and image repositories. The main feature is the ability to explore image and layer metadata without pulling the filesystem portion.

https://github.com/projectatomic/skopeo
 

Hubblestack is a security auditing tool build on top of SaltStack. It provides tools for file integrity scanning, integration with osquery and a reporting engine.

http://hubblestack.io/


Fabio is a fast, modern, zero-conf load balancing HTTP(S) router for deploying microservices managed by consul. Also provides integration with AWS ELB and API Gateway, and natively outputs graphite metrics.

https://github.com/eBay/fabio


Test Kitchen is a popular acceptance testing framework for infrastructure, and kitchen-terraform provides a plugin using Terraform to provision the infrastructure under test.

https://github.com/newcontext/kitchen-terraform


In pursuit of everything as code, Mermaid is a DSL for generating a wide variety of charts and diagrams, from gantt charts and flowcharts to sequence diagrams and more. It provides a powerful online editor and output to SVG too.

http://knsv.github.io/mermaid/index.html#mermaid

Remote management of servers is a common problem, and Chopsticks is a new solution. Rather than focus on a command line interface and a data language, Chopsticks is just a Python library.

http://mauveweb.co.uk/posts/2016/07/chopsticks.html
https://github.com/lordmauve/chopsticks


Cli53 is a handy command line tool for managing Amazon Route53 DNS records. As well as the ability to list, create and delete records you can also import and export from BIND formatted records.

https://github.com/barnybug/cli53

Configuring SSL is complicated. The TLS Observatory project provides a useful scanning tool which has the useful feature of providing recommendations for things to change.

https://github.com/mozilla/tls-observatory

  There's been an explosion of tools recently for bootstrapping a Kubernetes or Docker cluster. Spread is one such tool, but with a few nice looking features around reproducibility.

https://github.com/redspread/spread


Entropy is a fault injection system for Docker based systems. Define a policy and point it at your docker endpoints to trigger failures of containers to aid with building resilient systems.

https://github.com/buildertools/entropy


If you already have a reasonable size AWS setup, adopting a tool like CloudFormation can mean a lot of re-work. Bellerophon aims to generate CF templates automatically from existing AWS resources by querying the API.

https://github.com/arminhammer/bellerophon

dploy is an interesting looking opinionated deployment tool that leverages DC/OS and GitHub.

https://medium.com/@mhausenblas/introducing-dploy-f112f6a6bbdf#.br5feah1g

SwarmKit is a new toolkit from Docker for orchestrating distributed systems. It includes primitives for node discovery, raft-based consensus, task scheduling and more.

https://github.com/docker/swarmkit


Cake is an open source and cross platform build automation system with a C# DSL, particularly handy for building NuGet packages.

http://cakebuild.net/
 
Snapcraft is the packaging tool for the snappy Ubuntu Core transactional packaging system. It allows for composing software from multiple other package formats into a single tree.

https://github.com/ubuntu-core/snapcraft


Torus is an open source project for distributed storage coordinated through etcd. It ships with a simple block-device volume plugin, but is extensible to more and is seeing lots of development at the moment.

https://github.com/coreos/torus


SQL is one of those powerful and widely used common tools. KSQL brings the power of SQL to Kubernetes, allowing for querying resources like pods and services using SQL.

https://github.com/brendandburns/ksql


For all the simplicity in the basic Dockerfile, following the current best practice can be tricky. Dockerize Me provides an opinionated, wizard-like, approach to building Docker images.

https://github.com/fiunchinho/dockerize-me


Rocker is another Docker build tool, in this case one which adds some new features to the standard Dockerfile - including tagging and pushing from within the file, mounting build-time volumes and attaching to the build process part way through to aid debugging.

https://github.com/grammarly/rocker


We often talk about the importance of holding a learning review or blameless postmortem after an incident, but how do you track those over time? Enter Morgue. A simple web application, plus some interesting Jira and IRC integration.

https://github.com/etsy/morgue


If you run SSH on a non-standard port you still run the risk of people discovering that port. So why not change the port every 30s and use TOTP codes to discover it?

https://github.com/benjojo/totp-ssh-fluxer


Managing the configuration of a complex piece of software can too easily fall back into manual, error prone practices. Monkey Ops is an application which aims to ensure your OpenShift setup is robust by finding and finding resources like Pods or DeploymentConfigs and randomly terminating them.

https://github.com/Produban/monkey-ops


jwtproxy is an application for  authenticating, and possibly authorizing requests between services. It implements JSON Web Tokens and provides a forward and reverse proxy configuration for verification.

https://github.com/coreos/jwtproxy


Distributedlog from Twitter is a high-performance, replicated log service, offering durability, replication and strong consistency as essentials for building reliable distributed systems.

https://github.com/twitter/distributedlog

JMESPath is a query language for JSON. It comes with multiple compatible implementations in different languages, all based on a solid specification and test suite. I can see multiple uses for this.

http://jmespath.org/


Knox is another secrets management server with support for key rotation, auditing secrets access and with a desire to be developer friendly.

https://github.com/pinterest/knox


I’m a fan of using Cucumber style tests for infrastructure acceptance. The varnishtest cucumber project provides a set of steps for testing your VCL configurations, some good examples in the README.

https://github.com/nstielau/varnishtest_cucumber


Unik is an interesting attempt to build a unikernel deployment platform. It currently supports C/C++, Go (targeting rumprun) and Java (targeting OSv) and deployments to AWS, vSphere and Virtualbox.

https://github.com/emc-advanced-dev/unik


Many organisations using the same tools adopt different workflows to accomplish similar results. This Chef plugin helps sync local cookbooks with a Chef server, based around a workflow that eschews cookbook versioning.

https://github.com/PagerDuty/pd-sync-chef
 
Binctr allows you to create fully static, unprivileged, self-contained, containers as executable binaries. Which is pretty neat once you unpack that description and look at the demos.

https://github.com/jfrazelle/binctr


Harbor is a new container registry server for Docker containers with a nod towards enterprise features like role based access control and auditing capabilities.

http://vmware.github.io/harbor/
https://github.com/vmware/harbor

Airflow is a system to programmatically author, schedule and monitor data pipelines. The idea is to describe pipelines as directed acyclic graphs using Python. Airflow also comes with some great looking visualisation tools too.

https://github.com/airbnb/airflow


Habitus is a build tool for Docker images. It builds on Dockerfile, allowing for multi-stage builds as well as simple secrets management and environment variable injection.

http://www.habitus.io/


Napalm, or Network Automation and Programmability Abstraction Layer with Multivendor support, is a suite of tools for driving network devices with code, in this case Python. SUpport is already impressive, with JunOS, EOS, NXOS, IOS and more on the list.

https://github.com/napalm-automation/napalm

BCC is a set of dynamic tracing tools for linux. Powered by BFS which is available in 4 series kernels, BCC has tools for I/O latency, new processes, slow disk operations and lots more.

https://iovisor.github.io/bcc/


A number of operating systems are experimenting with different, and automatic, update mechanisms. CoreRoller is a set of tools to control and monitor the rollout of your updates, aimed at being an open source alternative to CoreOS CoreUpdate.

https://github.com/coreroller/coreroller


CFSSL is a toolkit for managing PKI/TLS. It contains lots of tools, including a certificate authority server that can use multiple signing keys, tools for managing certificate pool bundles and for generating private keys and CSRs.
https://github.com/cloudflare/cfssl


VIC, or vSphere Integrated Containers, is container runtime aimed at allowing developers familiar with Docker to develop in containers while allowing management via the familiar vSphere UI.

https://github.com/vmware/vic

Mas is a simple command line interface for the Mac App Store. Designed for scripting and automation.

https://github.com/argon/mas


Teleport is a tool for remotely accessing isolated clusters of Linux servers via SSH or HTTPS. It supports a number of team focused features as well as support for 2 factor authentication and auto-expiring keys.

https://www.lvh.io/posts/introducing-teleport.html
https://github.com/gravitational/teleport


An interesting new DNS server, CoreDNS is a fork of the Go based Caddy server which allows for writing middleware to influence DNS queries.

https://miek.nl/2016/March/18/announcing-coredns/
https://github.com/miekg/coredns
 

Libretto is a new Golang library to create Virtual Machines (VMs) on any cloud and Virtual Machine hosting platforms such as AWS, Azure, OpenStack, vSphere, or VirtualBox.

https://www.apcera.com/blog/using-golang-virtual-machine-management
https://github.com/apcera/libretto


Dalmatiner is a new metrics database that is fast, low overhead, and easy to query and manage. It uses ZFS under the hood for checksums, compressions and caching.

https://dalmatiner.io/
https://github.com/dalmatinerdb/dalmatinerdb


Kube2Consul is a simple bridge between Kubernetes and Consul. It will watch the kubernetes API for changes in Services and then register those Services in Consul.

https://github.com/Beldur/kube2consul


Blaz is a tool for bundling up you scripts into docker containers. It provides a simple Python based API, and allows for nested execution in separate containers too.

https://github.com/amiorin/blaz


Juttle is described as a dataflow programming language. Along with the language are an entire ecosystem of tools for using it to interpret data from graphite, elasticsearch, slack, AWS, various databases and more. Interesting to see more examples of how this fits into a monitoring setup.

https://github.com/juttle/juttle
https://github.com/juttle/juttle-engine

CPUStat describes itself as a “fancy top”. It’s an interactive tool for analysing CPU utilization and saturation in a helpful way, focusing on taking highly granular samples every 200ms.

https://github.com/uber-common/cpustat
 

DbDat is a tool for checking the security of your database, in particular working out which CVEs the database in question may be affected by. It has support for scanning everything from Postgres to Oracle and DB2 to Mongo.

https://github.com/foospidy/DbDat

Solo5 is a new base for unikernels, specifically providing a base to run other stacks on QEMU/KVM. The repository contains an example of running Mirage under KVM rather than Xen.

https://developer.ibm.com/open/2016/02/10/solo5-unikernel-low-level-programming-produces-high-return/
https://github.com/djwillia/solo5


Linkerd is described as a Modern RPC proxy for microservices. It uses Finangle and supports runtime traffic routing, load balancing, service discovery and built-in instrumentation,

https://blog.buoyant.io/2016/02/18/linkerd-twitter-style-operability-for-microservices/
https://linkerd.io/
https://github.com/BuoyantIO/linkerd

Teletraan is a new application deployment tool which provides support for zero downtime deploy, rollback, staging, continuous deploy; and many convenient features such as showing commit details and comparing different deploys.

https://engineering.pinterest.com/blog/open-sourcing-teletraan-code-deployment-system
https://engineering.pinterest.com/blog/under-hood-teletraan-deploy-system
https://github.com/pinterest/teletraan


Maltrain is a malicious traffic scanning utility, designed to run on all your nodes and collect information about potential malware of other problems revealed by looking at the network traffic. The documentation is excellent, with lots of clear examples.

https://github.com/stamparm/maltrail


Inductor is a command line tool used in conjunction with packer-windows to create Windows Vagrant boxes.

https://github.com/joefitzgerald/inductor


A demonstration of using a base image for running rumprun unikernels as Docker containers using KVM, fully integrated with Docker networking.

https://github.com/mato/docker-unikernel-runner

Simple tool for talking the contents of a configuration file and saving it as a Kubernetes secret for use by other Kubernetes resources.

https://github.com/kelseyhightower/conf2kube


Uphold is an interesting looking tool intended to verify backups. It downloads the backup, decompressing, loading and then running programmatic tests against it.

https://github.com/forward3d/uphold

 Builds servers can become complex quickly, sometimes because the problem you’re trying to solve is itself complicated. But if you have highly opinionated (git and GitHub only) and simpler CI requirements then Surf might be worth a look.

https://github.com/surf-build/surf


Goad is a nice demonstration of building a useful tool using Apache Lambda. It’s a Go based distributed load testing tool that takes advantage of the serverless architecture of Lambda to spin up on demand.

https://goad.io/


Keeping up with what service is running on what port is always tricky, and the commands to get the information tend to be somewhat cryptic. Enter whatportis, a simple interface for getting the ports in use by a service, or the service running on a specified port.

https://github.com/ncrocfer/whatportis

The debate about init systems and PID 1 inside a container has spawned a number of projects. S6 overlay looks particularly interesting, and makes some good arguments about why one thing, rather than one process, per container is a useful distinction.

https://github.com/just-containers/s6-overlay

ShellCheck is a static analysis tool for shell scripts. As well as providing hints on syntax it can also pick up on unused variables and in some cases suggest corrections for typos.

https://github.com/koalaman/shellcheck


Building on ShellCheck, Hadolint is a linter for Dockerfiles. As well as providing information based on Docker published guidelines it also picks up on issues with shell code using in RUN directives.

https://github.com/lukasmartinelli/hadolint


If you’re already familiar with Docker, but interested in runc, then the riddler tool is likely useful. It can inspect a running docker container and automatically generate the config and runtime files required to start the same container under runc.

https://github.com/jfrazelle/riddler
 
Minimesos is a set of tools to make running a Mesos cluster locally easy, even on OSX. Particularly useful if you’re developing your own mesos frameworks.

http://container-solutions.com/minimesos-0-5-0-mac-os-x-support-and-improved-docker-support/
https://github.com/ContainerSolutions/minimesos


Apex is a CLI tool for working with AWS Lambda, providing simple deployment, build and introspection capabilities and supporting Go and Python as well as the native Node.js.

https://github.com/apex/apex


Caddy is a new web server with out-of-the-box support for HTTP/2, IPv6 and WebSockets. It also has nifty built-in support for Let’s Encrypt to provide HTTPS by default.

https://caddyserver.com/
https://github.com/mholt/caddy


A nice idea for providing minimal help documentation focused on typical examples for common unix tools. A bit like a community contributed and stripped down man.

http://tldr-pages.github.io/


Pash is an Open Source reimplementation of Windows PowerShell, for Mono.

https://github.com/Pash-Project/Pash

A unikernel experiment, this time attempting to use Clojure but compiling down to Javascript and running on runtime.js under QEMU.

https://github.com/txus/cljs-on-unikernel-demo
 
AWSspec is a set of rspec matchers for testing AWS. The library supports a wide range of AWS resources and has nicely documented matchers for testing everything from whether an EC2 instance is in the correct VPC to whether an autoscaling group has been setup with a load balancer.

https://github.com/k1LoW/awspec


Containerd is daemon with an API and a command line client, intended to manage runc based containers on single machine. It from Docker, aimed specifically at high density and performance.

https://containerd.tools/


Git-appraise is a git plugin which implements a code review workflow for distributed teams, storing the review data in the fit repository itself.

https://github.com/google/git-appraise

With the public nature of GitHub and other repositories leaking passwords and other credentials is becoming a bigger problem. Enter git-secrets, which uses local git hooks to prevent you pushing code with know passphrases and other sensitive information in.

https://github.com/awslabs/git-secrets


Most systems administrators need to quickly find things stored in CSV files at some point. Q looks like a handy tool to do just that using a subset of SQL.

https://github.com/harelba/q

Snap is a new tool for gathering of telemetry from systems, with a focus on being particularly modular and easy to administer. The accompanying blog post goes into some of the why and where the project is going.

https://github.com/intelsdi-x/snap
http://nickapedia.com/2015/12/02/what-if-collecting-data-center-telemetry-was-a-snap/


There are always risks with a rapid shift in technology, and in normally slow moving organisations adopting fast moving technology. This cost looks a containers and picks out several concerning trends or gaps in understanding which could lead to problems.

http://www.juliandunn.net/2015/12/04/the-oncoming-train-of-enterprise-container-deployments/


Documentation is an important part of any internal or external application. Slate is a documentation generator focused on API documentation. It provides functional and attractive templates and works well on mobile too.

https://github.com/tripit/slate


Herokuish is a tool for bringing parts of the heroku workflow to containers. Specifically it allows for use of buildpacks and the Procfile with Docker, and aims to be a low level component for others to reuse that require this functionality.

https://github.com/gliderlabs/herokuish


The Docker event stream is underappreciated in my experience, giving you a stream of everything happening in Docker. Ahab is an interesting new tool for processing this stream in Python, the blog post gives an example of using it for dynamic assignment of IP addresses.

https://tech.instacart.com/ahab/
https://github.com/instacart/ahab

Wagl is a new tool for service discovery via DNS intended for use in Docker Swarm clusters.

http://ahmetalpbalkan.github.io/wagl/
https://github.com/ahmetalpbalkan/wagl


Sup describes itself as “make for network servers”. It allows you to describe a set of hosts and commands, and provides a simple CLI to interact with them. Similar to Fabric in Python, but in this case written in Go with a YAML configuration file.

https://github.com/pressly/sup


The King’s Hand is a tool for organizing ad-hoc scripts written in Go. It provides a simple discovery mechanism and a way of loading arbitrary go binaries into the kh tool.

https://github.com/bryanwb/kh
 
Facette is a new and nice looking time series and graphing engine, featuring integration with RRD, Graphite, Collectd and more.

https://facette.io/


AppCatalyst looks an interesting local development tool. Billed as a desktop hypervisor for developers, it provides a simple Photon OS based image for launching Docker containers for local development.

http://getappcatalyst.com/


Kurma is another minimal OS, built on the notion that everything is a container. It also provides an API for use in managing containers across a cluster of hosts.

https://github.com/apcera/kurma
https://github.com/apcera/kurmaos


Clair is a container vulnerability analysis service. IT aims to provide vulnerability information about containers based on Ubuntu, Debian and Centos - doing so using published upstream vulnerability information.

https://github.com/coreos/clair


Ansible-dk is an omnibus package providing Ansible and a number of related tools for developers, like ansible-lint and test-kitchen, intended to make getting started with a consistent environment easier.

https://github.com/omniti-labs/ansible-dk


Inspec is a new infrastructure testing tool based on the rspec Ruby testing framework. It has some nifty ideas, from a solid separation of transport to simple custom resources and dedicated CLI tool.

https://github.com/chef/inspec


A handy tool for anyone using Redis in a clustered setup. Redis Cluster Tool is a CLI app for checking the status of the cluster, rebalancing and getting and setting configuration data.

https://github.com/deep011/redis-cluster-tool


If you’ve ever wanted to use Ruby natively in your shell, instead of Bash, then Rush might be for you. Some nice examples and rationale.

https://s-mage.github.io/2015/10/25/rush.html
 
Helm is an interesting new project which aims to provide a package manager for Kubernetes, along with a shared community repository modelled after homebrew called Charts.

https://github.com/deis/helm
https://github.com/deis/charts


Sops is an editor of encrypted files that supports YAML, JSON and TEXT formats and encrypts with AWS KMS and PGP. Another useful tool for secrets management with excellent documentation.

https://github.com/mozilla/sops


Credstash is another tool for secrets management, specifically making use of AWS Kye Management Service but providing a nice clear user interface.

https://github.com/fugue/credstash


Snyk is a new tool aimed at discovering and fixing security vulnerabilities in third party packages included in your node.js project.

https://github.com/Snyk/snyk


Vegur is the newly opensourced reverse proxy library in use by Heroku. Provided as an Erlang library, this means you can implement custom routing yourself as a callback.

https://github.com/heroku/vegur


An interesting experiment in writing a threat model in code, alongside the application code. Threatspec parses comments in a specific format and outputs useful documentation and diagrams.

http://threatspec.org/
https://github.com/pki-io/threatspec


Anycast is commonly used in datacentres but generally not available in typical cloud environments like AWS. AWSnycast aims to address that by creating a routing daemon for AWS route tables. It can already be used to create an HA NAT service.

https://github.com/bobtfish/AWSnycast

An alternative user interface for Mesos. It provides a nice overview of cluster resources, insight into individual frameworks running and access to the master logs all from a web browser.

http://capgemini.github.io/devops/mesos-ui/
https://github.com/Capgemini/mesos-ui

Gryffin is a security scanning tool out of Yahoo that’s aimed to work at scale. The deduplication of tests sounds very useful for getting results quickly with testing identical pages.

https://github.com/yahoo/gryffin


Syntribos is another security scanner, but one specifically geared not to HTML and web pages, but to testing APIs.

https://github.com/rackerlabs/syntribos


A nice little dashboard that shows the number of commits on a repo since the last deploy. Surfacing this information can help avoid large deploys later.

https://github.com/dsingleton/deploy-lag-radiator


Amazon’s ECS service automatically allocates containers to hosts, which is great until you want to better understand your utilisation and where software is running. Enter Cloud Container Cluster Visualizer.

http://blog.mattcallanan.net/2015/10/c3vis-finding-your-docker-containers-on.html
https://github.com/ExpediaDotCom/c3vis


Automating your local development environment is useful, both for learning new tools and in any growing team environment. Superlumic is a new tool in that space that’s a thin wrapper around Ansible, which comes to some example configurations.

https://github.com/superlumic/superlumic
https://github.com/superlumic/superlumic-config
 

Muxy looks like a pretty comprehensive tool for testing network activity, by introducing delays, bandwidth constraints or other issues. It works as a simple proxy with a configuration file to control the features.

https://github.com/mefellows/muxy


Traefik looks an interesting new reverse proxy with some modern capabilities like integration with service management frameworks, hot reloading of configuration, exposing metrics over HTTP and an API for configuration.

https://github.com/EmileVauge/traefik


At this point the AWS interface is huge, so SAWS might be a good alternative command line interface. In particular it provides command completion and shortcuts which should make things both quicker and more discoverable.

https://github.com/donnemartin/saws

System XVI (or S16 for short) is a new modular init system which is currently under heavy development. Might be one to watch.

https://github.com/ServiceManager/ServiceManager

Whether the contents of a container present a security risk has been much discussed, and we’re starting to see tools emerge to help answer and automate parts of the problem. This example uses OpenSCAP to scan Docker containers, though this requires a SCAP profile for your chosen OS/image.

https://github.com/OpenSCAP/container-compliance


Dinit is a nice looking solution to running multiple processes inside a Docker container. It’s not a full init system which keeps things simple, and aims to fit well into a typical Docker build process.

https://github.com/miekg/dinit


Applikatoni looks like a nice, simple deployment server and provides a web based dashboard and CLI for a team to trigger and see software deployments.

http://applikatoni.com/


rtop-bot is a simple chat bot for bringing basic monitoring data to your Slack or HipChat room. Under the hood it simply SSH’s into machines to collect data on load, CPU and memory usage.

http://www.rtop-monitor.org/rtop-bot/

The problem of building Docker containers is definitely ripe for new tools to emerge, and dockramp looks interesting - it extends the Dockerfile syntax, makes smarter decisions about the creation of layers and aims to be faster than the standard docker build command.

https://github.com/jlhawn/dockramp


Many infrastructures will use an SSH jump box or two, rather than exposing all machines directly. SSHmuxd is an application which aims to make this easier to manage, at the same time as introducing some basic access control features.

https://github.com/joushou/sshmuxd


An interesting take on testing Puppet code, Puppet Spec uses the Puppet language to write tests for Puppet by adding a new assertion type to the language.

https://github.com/jolshevski/puppet-spec


Cloudsploit Scans is an interesting set of checks for scanning your AWS setup for potential security issues.

https://github.com/cloudsploit/scans

Hyper is described as a hypervisor agnostic docker engine. It’s able to launch tiny vms using the docker interface, providing the isolation of hardware virtualisation with the speed and user interface of containers. Very interesting.

https://hyper.sh/


Portus is  nice-looking user interface for the self-hosted Docker registry which also provides a team centric authorization service, so different teams can have there own namespace in the registry.

https://github.com/SUSE/Portus


Jungle is the start of a higher-level command line interface for AWS, currently supporting EC2 and ELB services.

https://github.com/achiku/jungle


Gotty is a handy application which exposes a command line application as a web application. It’s surprisingly configurable too.

https://github.com/yudai/gotty


Testing infrastructure is one of my pet topics, so these cucumber steps for openshift looked interesting. Some good examples of how to use them to test your applications are able to be installed and run.

https://github.com/vbehar/openshift-cucumber


Mantl is a distribution of software for running a container based infrastructure. It aims to integrate Mesos, Docker, Kubernetes, Calico and Logstash amongst other tools.

http://mantl.io/


Another interesting high-level distribution, Trusted Analytics integrates Cloud Foundry with various parts of the Hadoop ecosystem and aims to create a platform-as-a-service for data analytics.

http://trustedanalytics.github.io/


Sellsword is an interesting looking tool to switch between different environments (say a set of environment variables) - handy if you support different clients or work on lots of separate projects.

https://github.com/bryanwb/sellsword

dlayer is a handy tool for anyone building several docker images. It produces statistics about the use of layers, including how many layers are reused and how much space is being take up.

https://github.com/wercker/dlayer


Battleschool is a development environment provisioning tool based on Ansible, similar to Boxen or Kitchenplan.

https://github.com/spencergibb/battleschool


Testing your systems with inputs designed to cause problems is a good idea, but requires a good list of bad inputs. This list of naughty strings should come in handy.

https://github.com/minimaxir/big-list-of-naughty-strings

Y10K is a tool to deploy Yum/RPM repositories and mirrors in your local environment using settings described in a simple Yumfile.

https://github.com/cavaliercoder/y10k


Terraframe is an interesting approach to using Terraform for describing your infrastructure, specifically it provides Ruby based DSL interface rather than HCL.

https://github.com/eropple/terraframe


An interesting approach to building an application using the latest AWS features. Specifically it combines Lambda, API Gateway and DynamoDB for a stack that requires no managed servers.

https://github.com/jaws-stack/JAWS

ActiveMQ Artemis is a multi-protocol, embeddable, very high performance, clustered, asynchronous messaging system. It provides high availability and wide range of clustering options out of the box.

https://activemq.apache.org/artemis/


Python has always had a hand in data analysis work, but mainly for smaller or medium sized data sets. Ibis aims to change that, with first class integration the Hadoop ecosystem, without compromises in functionality, usability, or performance.

https://github.com/cloudera/ibis
http://www.ibis-project.org/


Racker is an opinionated Ruby DSL for generating Packer JSON templates. The main advantage appears to be composition and reuse, handy if you’re maintaining a number of templates.

https://github.com/aspring/racker


Testinfra is a tool for writing unit tests for infrastructure, similar to Serverspec but implemented in Python. It supports a few standard privatives like files, services, packages and users, as well as featuring integration with Salt and Puppet.

https://github.com/philpep/testinfra
http://testinfra.readthedocs.org/en/latest/

Hygieia is a single, configurable, easy to use dashboard to visualize near real-time status of a continuous delivery pipeline.

https://github.com/capitalone/Hygieia


Waffles is intended to be a simple configuration management and deployment system written entirely in Bash for portability.

http://waffles.terrarum.net/
https://github.com/jtopjian/waffles


Pupa is a handy bootstrapping tool for an opinionated modern Puppet setup using the latest puppet-agent as well as r10k to manage environments.

https://daenney.github.io/2015/07/23/Pupa.html
https://github.com/daenney/pupa

Apache nifi is a tool for data flow management. It provides a GUI for creating flows which can deal with system failure, overflowing data and more. The site also features comprehensive documentation on the capabilities and a good introduction.

http://nifi.apache.org/


SwifttypeMonitoring is an interesting looking set of Ruby helpers to help build application-level monitoring checks. with build-in examples of checkings for MySQL, ElasticSearch and Redis it’s relatively easy to see how to write your own checks.

https://github.com/swiftype/swiftype-monitoring

Aeron looks like an interesting new messaging system intended for very high throughput and very low latency. It’s described as a message oriented replacement for TCP rather than a general purpose messaging platform.

http://highscalability.com/blog/2014/11/17/aeron-do-we-really-need-another-messaging-system.html
https://github.com/real-logic/Aeron

Notary is a tool designed to allow people to publish and verify content. Based on the Tor update framework model, one of the usecases is to allow for the current curl sudo bash install model but with a verification model.

https://github.com/docker/notary


Image sprawl can be a problem with Docker and Sherdock aims to help.  Sherdock is a simple image management tool which features garbage collection of unused images as well as identifying orphaned volumes and other cleanliness-centric features.

http://rancher.com/sherdock/
https://github.com/rancherio/sherdock

Mesos-consul automatically registers/deregisters services run as Mesos tasks, so you can use consul for service discovery purposes in a Mesos cluster.

https://github.com/CiscoCloud/mesos-consul


Brubeck is a statsd compatible stats aggregator written in C. The initial post explains the problems with statsd, and in particular the use of UDP for monitoring.

http://githubengineering.com/brubeck/
https://github.com/github/brubeck


CollectdWin a system statistics collection daemon for Windows, inspired by 'collectd'.

https://github.com/bloomberg/collectdwin

Summon is a command-line tool that reads a config file and injects secrets as environment variables into any process. It’s intended to use different providers for the actual secret storage, and currently supports Conjur, S3 and OSX Keychain.

http://blog.conjur.net/introducing-summon
https://conjurinc.github.io/summon/


Security Content Automation Protocol (SCAP) is a way of describing various security and compliance controls. The Foreman project has just added an interesting integration, meaning you can run automated vulnerability assessment and security compliance audits across Foreman managed hosts.

http://foreman-shlomizadok.rhcloud.com/2015/06/09/openscap-foreman/

I’m a big fan of sysdig as an all-in-one tracing tool. Csysdig is a new, interactive, command line user interface to sysdig. It supports querying live data as well as looking through previously captured traces.

https://sysdig.com/announcing-csysdig-strace-htop-lua-container-support/


A nifty tool for sending alerts (to HipChat, PagerDuty, email, Slack, Influxdb) based on changes in Consul health checks.

https://github.com/AcalephStorage/consul-alerts

Tipboard is an open source dashboarding framework. It’s specifically designed for large screens and deals with all the display issues so you can just focus on the data.

http://allegrotech.io/tipboard/
https://github.com/allegro/tipboard


An interesting attempt to build a human readable workflow language for operations. Allows for writing reusable operations and then tying them together with flows.

http://www.cloudslang.io/
https://github.com/CloudSlang


EGADS (Extendible Generic Anomaly Detection System) is designed to automatically detect anomalies in large scale time-series data. Looks well suited to integration with some monitoring systems.

http://labs.yahoo.com/news/announcing-the-open-source-of-egads-a-scalable-configurable-and-novel-anomaly-detection-system/
https://github.com/yahoo/egads

Percheron is a command line and YAML based tool for describing a stack of related Docker containers.

https://github.com/ashmckenzie/percheron


A handy command line tool for interacting with the Marathon API. The README shows a simple text format example for describing applications running in Marathon.

https://github.com/eBayClassifiedsGroup/marathon_deploy

Terraframe is a Ruby based DSL for Terraform, giving you the power of a full programming language with the Terraform support for different cloud providers.

http://edcanhack.com/2015/01/terraframe-making-hashicorps-terraform-awesome/
https://github.com/eropple/terraframe

Chaos HTTP Proxy is a simple HTTP proxy which can be configured to break in ways useful to testing distributed systems. Mess with MD5 hashes, add client or server timeouts, throw in redirects, reorder response headers and return error codes.

https://github.com/bouncestorage/chaos-http-proxy


Scope, from Weaveworks, is a new tool for visualising how your growing container based infrastructure connects together, focusing on how the network reveals your architecture.

http://blog.weave.works/2015/05/13/announcing-weaveworks-scope-container-visibility/
https://github.com/weaveworks/scope


Swaddle is another take on building a single tool for packaging, whether you require debs, rpms or tars. It handily builds website documentation for each created package, supports signed packages and builds repositories as well as individual packages.

https://github.com/raphaelcohn/swaddle
 

Apollo is a collection of tools (Mesos, Weave, Docker, Consul, HAProxy and more) integrated together to allow for building your own platform-as-a-service.

https://github.com/Capgemini/Apollo


Nulecule is a work-in-progress specification for an open format to describe multi-component applications, similar to Docker Compose. I’ve included a link to an example manifest too.

https://github.com/projectatomic/nulecule
https://github.com/projectatomic/atomicapp/blob/master/examples/wordpress/nulecule

I talked about secret and key management being a common problem a few weeks back. Vault is another entrant into the space with a few nice features, including clear audit logs, rolling credentials and support for multiple storage backends.

https://www.vaultproject.io/


Disque is a an in-memory, distributed job queue. Its goal is to capture the essence of the "Redis as a jobs queue" use case. Currently described as alpha quality, but interesting to keep an eye on.

https://github.com/antirez/disque


CloudRouter is a Linux based full-stack SDN implementation including OpenDaylight.  It includes the features of traditional hardware routers, as well as support for emerging technologies such as containers and software-defined interconnection.

https://cloudrouter.org


Captain is a simple opinionated workflow for building, testing and publishing containers.

https://github.com/harbur/captain

Bottled Water is an interesting looking tool for streaming data from PostgreSQL to Kafka. You could then use that stream for warming caches, building a search index or doing any form of stream processing.

http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of-postgresql-and-kafka/
https://github.com/confluentinc/bottledwater-pg


Following on from last weeks tools for secret management, Sneaker is a tool for storing secrets on S3 using Amazon Key Management Store. Excellent documentation describing various different threat models too.

https://github.com/codahale/sneaker


A nice example of an opinionated container, it provides an Nginx HTTP load balancer which updates automatically based on information from etcd. It also integrates with another container running registrator to do this automatically.

https://github.com/factorish/proxy
 

Sometimes you just want a lo-fi dashboard. A browser window to refresh and reload as needed. Page Refreshr solves that problem nice and simply.

https://github.com/Jimdo/page-refreshr


Smolder is a simple tool for testing API endpoints. Describe the URLs, headers and required responses in JSON and run tests against your API. I’d probably prefer something more human

https://github.com/sky-shiny/smolder


Vega is a distributed mailbox, with an HTTP API and plans for AMQP and STOMP support. It’s designed to be durable and resilient under failure, and uses Consul under the hood. It’s intended as the backbone for how other components communicate with each other.

https://github.com/vektra/vega


Hub release is a handy utility for automatically populating release notes on github based on commit messages and closed issues.

https://github.com/zestia/hubrelease

As containers become the unit of work in distributed systems, scheduling across hosts becomes increasingly important. Citadel is interesting because, rather than being just an out of the box solution it’s positioned as a toolkit for writing your own custom schedulers.

http://citadeltoolkit.org/
https://github.com/citadel/citadel


Shpec is an rspec-like testing framework for shell scripts. The README gives you an idea of the syntax, and show how easy it is to extend with your own matchers.

https://github.com/rylnd/shpec


A simple Rake library which allows building Debian packages using either git-buildpackage or fpm. It would be nice if this supported other package types like RPM as well.

https://github.com/jimdo/buildtasks

 


Graphite beacon is a very simple alerting system for Graphite metrics. Just install the app and describe your alerts in JSON. Can alert via SMTP, Hipchat, Slack or HTTP.

https://github.com/klen/graphite-beacon


A nifty little tool, httpdiff allows you to diff two HTTP requests and show any differences in the headers. Useful for exploration or building testing tools on top of.

https://github.com/jgrahamc/httpdiff


Bazel is a new build tool from Google, it’s able to build a wide range of software quickly and is particularly suited to organisation using a single very large monolithic repository.

http://bazel.io/

A nice and simple way of getting metrics from elasticsearch to graphite. Nice screenshots to show you what you’re getting too.

https://github.com/jamiealquiza/es-stats


A minimal githook application configured using a json file. You can set commands to run based on the repository changing and sent output to stdout and to slack.

https://github.com/octplane/small-deployer

Shutit is a build tool for Docker containers, moving from the Dockerfile syntax to using python as a build language and providing an opinionated workflow.

http://ianmiell.github.io/shutit/
https://github.com/ianmiell/shutit

A new CI system with a very pipeline native setup. It’s automatically deployed using BOSH and features some interesting integration with Pivotal Tracker, Docker, S3 and more.

http://concourse.ci/


A collection of Puppet modules and scripts aimed at compliance with more than 1000 requirements from nine US Department of Defence policy documents. Some interesting autogenerated documentation with Latex which might be applicable to others too.

https://github.com/afseo/cmits


The combination of low level system tools with embedded scripting with Lua is, I think/hope, one we’ll see more of. Nginx already does this well. go-lua is a Lua runtime for Go, which I’d love to see in the emerging Go system tools.

https://github.com/Shopify/go-lua


Curl is amazing, but it’s also somewhat cryptic. Httpie is a a Curl replacement designed with a more friendly, and HTTP native, interface.

https://github.com/jakubroztocil/httpie
 

RancherOS is a minimal Linux distro designed to run Docker containers. It basically runs Docker as PID 1 a top the Linux kernel, meaning the image itself is tiny compared to a normal distribution.

http://rancher.com/rancher-os/


Packetbeat is an open source application monitoring and packet tracing system. It works by sniffing the traffic between your application services, correlating them into transactions and using Elasticsearch for analysing them and for ad-hoc queries.

https://github.com/packetbeat/packetbeat


The data in AWS CloudTrail is essential for security and compliance in AWS. Traildash grabs those files, stores them in ElasticSearch, and presents a Kibana dashboard so you can analyze recent activity in your AWS account.

https://github.com/AppliedTrust/traildash


Burlesque is a message queue with a long polling HTTP API for easy integration. The excellent documentation gives a rundown of how to use it and the various configuration options.

https://github.com/KosyanMedia/burlesque

A very good looking dashboard for managing a fleet of Apple computers using Munki.

https://github.com/salsoftware/sal


Gogs is a self-hosted Git server. It’s aim is to be cross-platform (running on Windows, Mac and Linux), light-weight and very easy to install.

http://gogs.io/


Baci is a tools for building ACIs, the Rocket container format. It handily reuses the Dockerfile format, but also speculates a little about a more general container build language which sounds interesting.

https://github.com/sgotti/baci

Project Jellyfish is a broker system. It allows admins to create a product catalog of any type of service (SaaS, IaaS, PaaS, etc.) and allows them to be assigned a cost, and then users can create projects and add those services to a project.

http://www.projectjellyfish.org/
https://github.com/projectjellyfish/api


Cram is a testing framework for command line applications. Cram tests look like snippets of interactive shell sessions which makes writing tests nice and easy.

https://bitheap.org/cram/


Building system packages always required a fairly good understanding of the packaging tool chain, and building packages for different operating systems required even more work. This project uses Docker under the hood to create reproducible RPM build environments nicely.

https://github.com/alanfranz/docker-rpm-builder

Debops is a collection of ansible playbooks and supporting scripts intended to make administering debian easier.

http://debops.org/
https://github.com/debops/debops

Comcast is a tool designed to simulate common network problems like latency, bandwidth restrictions, and dropped/reordered/corrupted packets. Very handy for testing distributed systems under non-ideal network conditions.

https://github.com/tylertreat/Comcast


Toxiproxy is another tool for simulating network errors for testing purposes. It provides a proxy with an API, which allows for altering the network conditions cleanly. The example gives you a great idea of what’s possible.

https://github.com/Shopify/toxiproxy

huptime is a clever utility to enable zero-downtime reloading of processes, without modifying the application. It supports running under init systems like upstart too.

https://github.com/amscanne/huptime


If you’re on Windows and prefer a bundled software over installing and managing lots of small pieces then Bills Kitchen might be for you. It provides Vagrant, Chef and a Ruby developent environment along with various Vagrant plugins, the Putty SSH client, the SublimeText editor and more.

https://github.com/tknerr/bills-kitchen


A comprehensive set of Puppet and Chef code for hardening servers. Includes code for base operating systems, ssh, mysql, postgresql, apache and nginx and supports Ubuntu, RedHat, Debian, Centos and Oracle.

https://telekomlabs.github.io/


Jetpack is a work-in-progress implementation of the Application Container Specification using FreeBSD Jail and ZFS.

https://github.com/3ofcoins/jetpack


Dashiell is a nice frontend to osquery and facter, allowing you to query the state of a cluster of servers very quickly.

https://github.com/maclennann/dashiell

Novm is an interesting take on a hypervisor, mixing some of the benefits of virtualisation with some of the advantages of containers. The README does a good job of explaining the rationale.

https://github.com/google/novm


An attempt to write a Chef client in PowerShell. Adds new Chef resources and Knife commands which use PowerShell and DSC on the backend.

https://github.com/POSHChef/POSHChef


Cachet is an open source status page system that aims to be simple to use and easy to manage. A good starting point if you’d prefer to run your own status page.

https://cachethq.io/


A simple application which generates a Vagrantfile suitable for common Ruby on Rails development tasks.

https://railsbox.io/
 

DeployDB is a tool for storing the state of deployments, and is designed to integrate with existing tools like Jenkins and RunDeck and actually do the deployment.

http://hackers.lookout.com/deploydb/
https://github.com/lookout/deploydb


RFC Reader does exactly what you expect, it provides a nice web interface for reading individual RFCs.

http://www.rfcreader.com/

Socketplane have released a technology preview of their container networking technology. The preview features things like ZeroConf multi-host networking for Docker, support for multiple networks and distributed IP Address Management (IPAM).

https://github.com/socketplane/socketplane


vCloud Air now has a nice looking open source command line interface. This post and repo cover the basic functionality.

http://blog.pacogomez.com/vcloud-air-command-line-interface/
https://github.com/vmware/vca-cli
 

Spigio is an interesting tool for modelling gossip based communications in large networks. The output to both GraphML and graphJSON make visualising results easy.

https://github.com/adrianco/spigo

Conn Check is a new tool for verifying software deployments, things like can  a service talk to a database or another service, or is authentication setup correctly. It can also auto-generate tests for some types of applications.

http://1stvamp.org/verifying-post-deploy-connections-with-conn-check


Dockerana is a packaged version of Graphite and Grafana, specifically targeted at metrics from Docker.

http://dockerana.com/


Seagull is another monitoring tool aimed at Docker environments. It provides a dashboard containing information about the images and containers, and allows simple actions like deleting an image or stopping a container.

https://github.com/tobegit3hub/seagull

Charted is a way of sharing graphs. I’m a big fan of using monitoring tools for business metrics, but exposing that data can be hard. Charted looks a great bridge to get other people interested in data.

https://medium.com/data-lab/introducing-charted-15161b2cd71e

Rancher is an attempt to bring AWS-like functions, such as EBS, VPC, ELB and Security Groups, and make them available across any servers, using docker under the hood.

http://www.rancher.io/
https://github.com/rancherio/rancher


VclFiddle is an online tool for experimenting with the Varnish Cache HTTP reverse-proxy in a sandboxed environment. The hosted version gives you a really easy way of debugging Varnish configuration.

https://github.com/vclfiddle/vclfiddle
http://www.vclfiddle.net/

Bosun is an all-in-one monitoring system backed by OpenTSDB. It provides alerting, time series data collection and visualisation as well as metric collection. It also treats both Linux and Windows as first class citizens.

http://bosun.org/


Octopush is another tool for using Jenkins for deployment automation. It provides a dedicated user interface and API, as well as integration with GitHub for authentication.

https://github.com/olx-inc/octopush
http://olx-inc.github.io/octopush/

Walter is described as a tiny deployment pipeline. IT reminds me of the wercker build file, but can run locally as well as within a CI system. Useful for testing pipelines locally.

http://ainoya.io/walter
https://github.com/Walter-cd/Walter/


Stager is a nifty looking tool to automatically build staging environments from GitHub pull requests.

http://info.localytics.com/engineering-blog/stager-staging-environments-on-autopilot
https://github.com/localytics/stager-server

osquery exposes information about your operating system via a SQL interface, allowing for some pretty powerful tooling. Also supports querying across multiple hosts.

https://code.facebook.com/posts/844436395567983/introducing-osquery/
https://github.com/facebook/osquery


Dwight Conrad is an OpenResty based HTTP routing component, backed by Redis. It supports a number of features useful for feature toggling and canary releases.

https://github.com/3scale/dwight-conrad


Cluppet is a tool to maintain a part of your AWS infrastructure in code, specifically as a clojure based DSL.

https://github.com/mixradio/cluppet

Stocker is an interesting approach to securely managing environment variables for use by applications, in particular containers. It’s currently Redis backed though could support multiple backends in the future.

https://github.com/buth/stocker
 

Boxstarter is a tool for bootstrapping a new Windows machine. It makes heavy use of Choclately packages and provides a number of very simple ways of getting started. Great documentation too.

http://boxstarter.org/


Carbonate is a set of command line tools for managing a Graphite cluster. It packages tools for listing and finding metrics, synchronizing metrics between nodes, backfilling whisper data from other files and more.

https://github.com/jssjr/carbonate


If you’re building software on GitHub and want an easy way of managing GitHub releases then look no further than this simple command line tool. The main features are around setting release metadata and uploading binary files as part of the release.

https://github.com/aktau/github-release

Taste tester is another workflow tool for Chef, it launches a Chef-Zero instance locally and then checks what changes would happen to a remote machine with your local changes.

https://github.com/facebook/taste-tester


A set of scripts useful for hardening a typical Linux system. Partly useful for learning about a good practices, I’d also love to see this converted into various config management code flavours.

https://github.com/marshyski/quick-secure

Chef Guard looks like a useful tool, specifically it protects your Chef server from untested and uncommitted cookbooks. Several useful features, including emailing of configuration change diffs and validation of changes.

http://xanzy.io/projects/chef-guard/introduction/overview.html


Learning new tools and languages is always hard so anything that makes it easier it a good idea. Chef Koans takes inspiration from other similar projects and applies it to Chef, a good starting point if you’ve done similar learning exercises before.

https://github.com/leftathome/chef-koans


Zonify is a set of command line tools for managing Route53 DNS for an AWS infrastructure. It intelligently uses tags and other metadata to automatically create the associated DNS records.

https://github.com/airbnb/zonify
 

The new puppet server was released at Puppet Conf. Puppet server is a drop-in replacement for the current puppet master with some pretty big performance and scalability improvements.

https://github.com/puppetlabs/puppet-server


If you’re using Route53 DNS then Roadworker should be of interest. It’s a Ruby based DSL for creating zones and records. Good examples too.

https://github.com/winebarrel/roadworker


If like me you’re maintaining a number of puppet modules you probably have an occasional day when you try and remember to update lots of static files (Rakefile, .travis.yml, Gemfile, etc.) across all of them. Modulesync looks a nice solution to that issue.

https://github.com/puppetlabs/modulesync

Blender is a modular remote command execution framework. It provides a few basic primitives to automate cross server workflows, with those workflows expressed in a ruby DSL and executed using the provided command line tools.

https://github.com/PagerDuty/blender


Mcrouter is a tool for those running large memcached clusters. It’s a memcached protocol router designed to shard data across a cluster of memcached instances, with uses for load balancing, cache distribution and a number of other failure cases.

https://github.com/facebook/mcrouter
https://code.facebook.com/posts/296442737213493/introducing-mcrouter-a-memcached-protocol-router-for-scaling-memcached-deployments/


Vitess is a project to provide servers and tools which facilitate scaling of MySQL databases for large scale web services. Comprehensive user documentation is provided if you want to try it out.

https://github.com/youtube/vitess


Urknall is an interesting looking remote execution framework written in Go. Much of the functions make use of standard shell commands, which makes getting started very simple.

http://urknall.dynport.de/
https://github.com/dynport/urknall

If you’re doing any real PowerShell work in the console that PSReadLine is for you. It provides a much nicer command line interface with completion, history, syntax highlighting, multiline support and lots more.

https://github.com/lzybkr/PSReadLine
http://rkeithhill.wordpress.com/2013/10/18/psreadline-a-better-line-editing-experience-for-the-powershell-console/


The concept of a single source of truth for everything pertaining to your infrastructure isn’t a new idea, but Collins looks like a more modern take on the CMDB. A documented API and client libraries, along with support for physical servers, switches, racks and higher level software properties.

http://tumblr.github.io/collins/


Portainer is an interesting combination of mesos and docker, aimed at building containers rather than running them. Worth investigating is you’re finding yourself building lots of containers and managing dedicated machines to do so.

https://github.com/duedil-ltd/portainer


Oh My Vagrant is an opinionated development environment suitable for docker, puppet and more development activities. Based around Vagrant, Fedora, libvert and more.

http://ttboj.wordpress.com/2014/09/03/introducing-oh-my-vagrant/


If you’ve found yourself writing small scripts in bash, and then re-implementing those same scripts for Windows as batch files then Batsh might be for you. A small scripting language that compiles to batch and bash. The online demo is pretty nifty.

https://github.com/BYVoid/Batsh
http://batsh.org/

If you’re using Test Kitchen then you might find the kitchen-sync plugin useful. It provides alternative file sync strategies which should be faster than the defaults for certain setups.

https://github.com/coderanger/kitchen-sync


An interesting DNS server which connects to the AWS API and returns DNS records based on instance names or roles.

https://github.com/ConradIrwin/aws-name-server


Scumblr is an interesting tool for keeping an eye on potential exploits or discussions of your company/application elsewhere on the internet. All sorts of potential monitoring applications for this.

https://github.com/netflix/scumblr
http://techblog.netflix.com/2014/08/announcing-scumblr-and-sketchy-search.html


Dockersh is a user shell for isolated, containerized environments and is designed to be used as a login shell on machines with multiple interactive users.

http://engineeringblog.yelp.com/2014/08/hack209-dockersh.html
https://github.com/Yelp/dockersh


Helios is another docker orchestration platform designed for managing containers across a large fleet. Instructive documentation, debian packages and a vagrant setup for trying it out are all plus points.

https://github.com/spotify/helios
 

Mentioned in the ftrace blog post above this set of scripts is a great resource if you’re doing any low-level performance tuning on Linux.

https://github.com/brendangregg/perf-tools


Vessel is another tool for automating the setup and use of docker based development environments. It integrates Vagrant, CoreOS, Consul, docker and Atom into a slick opinionated package. The user interface looks particularly impressive from the screenshots.

http://awvessel.github.io/


Ganger is a tool for running short-lived network services inside Docker containers, and then proxying client connections to them. It’s designed mainly for data processing batch jobs that require a fresh environment each time, but could have other interesting uses.

https://github.com/andytinycat/ganger


Chef-runner is a tool for executing chef recipes on remote machines, over SSH. It features vagrant integration and is built to be much faster than the in-built provisioning for this use-case.

https://github.com/mlafeldt/chef-runner

Cloud Foundry is a pretty complex distributed system, and running the entire thing locally when all you want to do is develop apps to run on it isn’t ideal. Enter Cloud Focker, which combines Docker with Cloud Foundry buildpacks under a nice command line interface.

https://github.com/CloudCredo/cloudfocker

 

Kibana-rack embeds the Kibana ElasticSearch interface into a Rack middleware, making running it alongside existing ruby apps simpler. The example showing authentication integration with Devise shows why this might be a good idea for some.

https://github.com/tabolario/kibana-rack


An interesting curated set of Go libraries for common operations and automation tasks. Everything from DNS and consensus protocols to load testing and metrics collection.

https://github.com/mindreframer/golang-devops-stuff


This packer provisioner allows for running provisioning command on the host rather than the new machine being built. I can imagine a few uses for this, from black box testing to recording the results of packer builds.

https://github.com/shaunduncan/packer-provisioner-host-command


Diplomat is an mechanism for integrating Ruby applications with the Consul service discovery tool. The blog post nicely covers some of the advantages of this approach.

https://github.com/WeAreFarmGeek/diplomat
http://johnhamelink.com/distributed-web-systems-with-consul-haproxy-and-envoy.html

Managing local machines with Vagrantfiles has lots of advantages, but when recreating that environment in production you might turn to something like libcloud. Duplicate no more, with with vagrant driver for libcloud.

https://github.com/carletes/libcloud-vagrant


Terraform is a tool aimed at giving you a single view of your entire infrastructure in code. Describe instances and supporting services (like DNS) via a domain specific language and use the command line tool to launch across multiple infrastructure providers.

http://www.terraform.io/


Consul is an orchestration mechanism with fault-tolerance based on the gossip protocol and a key/value store that is eventually consistent. Hiera-consul will allow hiera to write to the k/v store for metadata centralisation and harmonisation.

https://github.com/lynxman/hiera-consul

Devstep is described as a development environment builder powered by Docker and buildpacks. The getting started docs give you a flavour of the workflow and a list of the current buildpacks.

https://github.com/fgrehm/devstep
http://fgrehm.viewdocs.io/devstep


Infrataster is another tool for writing unit tests for infrastructure. It’s designed to complement serverspec, by focusing on running tests from outside the virtual machine to test external interfaces like HTTP or SSH.

https://speakerdeck.com/ryotarai/infrataster-infra-behavior-testing-framework-number-oedo04
http://infrataster.net/


Uchiwa looks like a great addition to the Sensu monitoring framework. The screenshots give you an idea about it’s capabilities. It’s also packaged as a docker container which I think is something we’ll see more and more of.

https://github.com/sensu/uchiwa


Goiardi is a Chef server written in Go with the ability to run entirely in memory, with optional persistence saving the in-memory data to disk or using MySQL or Postgres. I’m not sure of the why in this case, but having open integration test suites make this an interesting possibility.

https://github.com/ctdk/goiardi


Another log parsing tool, Petit has a number of impressive features from hashing, graphing and discovering important repeated items in log files.

http://crunchtools.com/software/petit/
 
MaestroNG is an orchestration tool for multi-host docker environments. The examples in the README give the best idea, showing YAML files describing the entire environment.

https://github.com/signalfuse/maestro-ng


Anode is a utility for analyzing graphite metrics. It’s currently quite experimental, containing a single three sigma analyzer that takes a time series from graphite and creates new metrics alongside.

https://github.com/mattrco/anode.exp


Log File Navigator is an advanced log file viewer for the small-scale. It’s only really useful for a single host but provides some impressive command line visualisation tools as well as a powerful SQL query engine.

http://lnav.org/


Haka is described as an open source security oriented language which allows to apply security policies on (live) captured traffic. It’s based on Lua and looks to be a great way of writing certain kinds of network tools.

http://haka-security.org/


Another interesting networking stack, again using Lua (LuaJit in this case). Snabb Switch is designed for high throughput, low latency network sytems suitable for use by ISPs. It speaks natively to ethernet hardware, hypervisors, and the linux kernel. Interesting times for networking.

https://github.com/SnabbCo/snabbswitch


An interesting approach to server authentication, swapping the AuthorizedKeysCommand for a custom application which checks keys published to GitHub rather than a local public key. The examples directory makes understanding how it works nice and easy.

https://github.com/davidpelaez/gh-keys


Tesera is yet another graphite frontend with a nifty way of separating the query from the visualisation, leading to some nice features. It looks gorgeous too with multiple colour themes available.

http://urbanairship.com/blog/2014/06/30/introducing-tessera-a-graphite-frontend
https://github.com/urbanairship/tessera


Another lovely looking dashboard, Puppet Explorer is a frontend for PuppetDB. It’s intended to exploring Puppet resource and node information with nice visualisations and the powerful PuppetDB query language.

https://github.com/spotify/puppetexplorer


Toaster is another tool designed for testing infrastructure stacks. It makes use of LXC under the hood to provide isolated test environments and comes bundled with a web interface to observe and manage test runs.

https://github.com/whummer/toaster


Given our daily use of version control systems they contain an awful lot of data past just the source code. This tool allows for exporting a git repository into the solr search engine for data mining.

https://github.com/arafalov/git-to-solr


I’ve mentioned OSv previous as an interesting take on the operating system, but trying it out locally had required a lot of effort. Enter capstan, which provides a very nice command line interface to launch OSv instances locally on your machine.

https://github.com/cloudius-systems/capstan


Cayley is an open source Graph database. It supports multiple storage backends, an HTTP based API as well as a REPL and a built-in query editor and visualiser.

https://github.com/google/cayley

If you’ve ever looked into ways of securing traffic between internal services you’ll have found lots of options, most of them lots of work. Certified is an internal CA that’s designed to be easy to use

https://github.com/rcrowley/certified

Kubernetes is the open source version of Google’s container cluster manager, including support for docker. The setup instructions are based around Google Compute Engine but the software can run anywhere.

https://github.com/GoogleCloudPlatform/kubernetes


Another announcement from DockerCon, Centurion is a deployment tool designed for docker. It provides a Rake based DSL which makes specifying hosts and images. The examples give you an idea, but the rolling update feature in particular is nice.

https://github.com/newrelic/centurion
 

Following on from a recent blog post about alert design, Nagios Herald is a notification application designed to help add context to monitoring alerts. The included examples should give you lots of ideas.

http://codeascraft.com/2014/06/06/introducing-nagios-herald/
https://github.com/etsy/nagios-herald


Logstash has always been very much about the plugins. This project aims to make writing those plugins much easier. Still early days but it looks to automate some of the common tasks.

https://github.com/coolacid/logstash-pluginbase



Changelog is a tool designed to answer the question “What changed in the last twenty minutes?”. It provides a simple HTTP api and a web interface and is intended for tracking everything from deployments, dns changes, reboots, creation of servers, etc.

http://engineering.prezi.com/blog/2014/05/28/changelog-a-tool-designed-to-help-you-recover-faster/
https://github.com/prezi/changelog


Openduty is an open source Alerting and incident escalation tool. It supports alerting with XMPP, email, SMS, Phone and Push notifications, and supports the Pagerduty API for triggering alerts.

https://github.com/ustream/openduty


Checkzilla is simple tool for letting you know about out-of-date software versions. It currently supports Rubygems and NPM, but it’s extensible for other package types. Being able to manage this in one place feels like a good thing.

https://github.com/mickey/checkzilla

Container agent is a tool for launching a set of containers based on a simple manifest file. Although designed for Google Compute Engine it appears like it should work anywhere. The documentation contains lots of good examples.

https://github.com/GoogleCloudPlatform/container-agent


Radial is described as being intended to help understand, record, and put into practice the intersection between Twelve-Factor App best-practices and Docker features. It’s provides a defined language as well as examples of how to compose a set of containers.

https://github.com/radial/docs


An OWASP project, Mantra is a browser for use when auditing the security of web applications. Under the hood it’s Firefox with a large number of very useful plugins, from traffic capture to XSS scanners.

http://www.getmantra.com/index.html

Packetbeat looks like a very nice new open source application monitoring and packet tracing system. A lovely Kibana based interface and agents which can detect various types of traffic once installed make for a very simple getting started experience.

http://packetbeat.com/


A neat approach to managing users in a large GitHub organisation. Provides tools for auditing users, ensuring two factor authentication is enabled and providing reporting of issues.

https://github.com/guardian/gu-who


Managing multiple different git hooks across multiple repositories for self-hosted git repos can be a pain. cpthook aims to make that easier, moving the configuration into a simple YAML file and providing a command line to manage the hooks.

http://cpthook.com/

I’ve been starting to think about more dynamic load balancing recently and projects like Vulcand are very interesting. It combining a custom HTTP Proxy with configuration stored in Etcd, with the configuration updating automatically as the config changes.

https://github.com/mailgun/vulcand


A very simple but powerful deployment user interface based around Git and a simple YAML configuration file.

https://github.com/gengo/goship


Kamon is a nice, opinionated, distribution of tools designed to make collecting metrics from scala applications easier. It uses Graphite or New Relic as the backend and provides modules for monitoring actors, tracing application requests and logging.

http://kamon.io/