Pages

Friday, 22 January 2021

Sken.ai DevSecOps walkthrough

 

Sken.ai Introduction:-

Sken.ai is a Continuous Application Security Testing and vulnerability scanning tool. It helps in static code analysis, Source code analysis, code review, code complexity, and penetration testing. It is a new and automated DevSecOps tool that provides multi-language scanning features in one place. It provides comprehensive Saas based continuous application testing software developers and DevOps, without the need for any security expertise. It helps to Find and fix all types of application security issues within your DevOps CI/CD cycle.

It provides a SAAS orchestration layer that integrates continuous application security testing into your DevOps CI/CD workflow automating the use of any open source security scanners across all scan types; SAST, DAST, SCA, MAST, container scanning, license compliance, and secrets detection. 


What will you learn from this article/blog:-

  1. How to provide security in your CICD pipeline?

  2. What is SAST/DAST/SCA/MAST and how it works?

  3. How sken.ai works? Which features and scanners sken.ai provides?

  4. What vulnerability scanning and penetration testing is?

  5. List of Top IDE’s used by developers provided by DevOps?

  6. List of version control tools provided by DevOps?

  7. List of Top CI/CD pipeline tools provided by DevOps?

  8. What web application security scanner is?


Who will need this tool:-

    There are lots of open-source application security testing tools available in the market. So it's very hard and confusing to select what exactly we need. Here automation comes further there are many open-source plugins and security tools are available to provide and scan our application for security tests before the development state goes to production.

Sken.ai provides automated scanning features and Generally, it is used to find out the bugs at early stages of the development of software (coding stage). With this code review, the quality of the software gets improved and the bugs/errors in the program code decrease. The Code Review Tools automates the review process which in turn minimizes the reviewing task of the code. It provides free scanning forever for single users, Unlimited tests on open source projects, 300 tests per month across all apps on private projects, Priority Scoring, and Continuous Monitoring. As it is a freemium (with about 300 free scans per month) it is the best choice for beginners and small project developers.





What does Sken.ai contain:- Here are some detailed points that I have discussed about the Sken.ai. It helps to scan every application in any language from any CI/CD Platform (such as Jenkins, Bamboo, etc). It uses all scan types SAST, DAST, SCA, Secrets, and more to scan at the coding time or the runtime. As I said before it enables security by simply adding a 2-line code snippet into CICD to enable the complete range of tests. It provides open-source scanners packaged and managed, There's No need to set up, install or update scanners. It selects & auto-discover scanners as per scanning context.


Sken.ai performs the most comprehensive security scanning and testing of applications during the CI/CD process. This includes not only static source code analysis (SAST) and software composition analysis (SCA), but also dynamic security testing (DAST) to uncover run-time security holes as well. This ensures that all the common avenues hackers use to execute security breaches are discovered before going to production. By providing the relevant scan type at the right stage in the DevOps and CI/CD stage, Sken.ai covers more vulnerability and compliance holes than anyone's capability alone. It uses all major development languages & frameworks including Java, PHP, Python, Microsoft .NET, NodeJS, Ruby, GO, and more.

   

Sken.ai applies machine learning and data science to normalize and prioritize all the application security issues found across testing categories. It provides AI-Based Analysis, and it reduces noise up to 97%. It Aggregated scan findings View and filter findings across multiple scans. It tracks bugs through code changes and provides standard risk ratings across all scan types. DevOps and AppSec teams can manage their entire bug fix and mitigation processes with Sken.ai, including prioritization and risk scoring, bug verification, task assignment through resolution. It is Intelligent & prioritized to fix critical findings first. It helps users to Manage findings from the Sken.ai portal.


Here are some Open Source Scanners included in Sken.ai internally:

  1. SCA: OWASP Dependency Leaks.

  2. SAST: Find-Sec-Bugs, NodeJScan, Brakeman, Bandit, Gosec, PHPCodeSniffer, ES Lint, TSLint.

  3. DAST: OWASP ZAP

  4. Secrets: Gitleaks, Trufflehog.







Static application security testing (SAST), or static analysis, is a testing methodology that analyzes source code to find security vulnerabilities that make your organization’s applications susceptible to attack. SAST scans an application before the code is compiled. It’s also known as white box testing. SAST takes place very early in the software development life cycle (SDLC) as it does not require a working application and can take place without code being executed. It helps developers identify vulnerabilities in the initial stages of development and quickly resolve issues without breaking builds or passing on vulnerabilities to the final release of the application.

A dynamic analysis security testing tool, or a DAST test, is an application security solution that can help to find certain vulnerabilities in web applications while they are running in production. A DAST test is also known as black-box testing, it is performed from the outside.


Traditional security scanners are proprietary, expensive, and difficult to use. They also need a team of app sec specialists to operate them, thus adding to the total cost. Sken.ai uses an innovative approach that uses a combination of popular, effective, and widely used open source security scanners to dramatically reduce the cost. Sken.ai provides free scanning forever for single users, Unlimited tests on open source projects, 300 tests per month across all apps on private projects, Priority Scoring, and Continuous Monitoring.


Vulnerability scanning is a continuous process that fits throughout an app’s software development life cycle (SDLC). Frequent scanning, especially in the early stages of development before upstream components are layered onto the source code, makes it much easier to fix issues before they become costly problems.
   

Sken.ai also provides IAST(Interactive Application Security Testing) Scanners are DAST scanners with some SAST capabilities. They test whether known code vulnerabilities can be exploited in the running app. Developers often layer API scanners over other tools, such as SAST and DAST, to ensure their APIs stay secure, even after a code or interface change. API scanning focuses on uncovering existing and potential security vulnerabilities like SQL Injection, Content-Type Missing, Misused Exception Handling, and Parameter Tampering.


For development, DevOps helps users to automate source code by providing IDE's like Microsoft Visual Studio, NetBeans, IntelliJ Idea, Eclipse, PyCharm, etc. For version control management Git, GitHub, GitLab, CVS, SVN, and for CI/CD pipeline DevOps provides Jenkins, CircleCI, Bamboo, Travis CI, etc. It enables security with a simple, 2-line code snippet to enable the complete range of tests. It provides Comprehensive testing across all scan types SAST, DAST, SCA, Secrets, and more. Perform security testing for all major development languages/frameworks. Sken.ai application developers are notified of security vulnerabilities as they commit and build their code. Also, Sken.ai helps app sec and DevOps respond to the vulnerability alerts, prioritize the issues found, and fix them during the code and build process.


A CI/CD pipeline is a series of steps that must be performed to deliver a new version of the software. Continuous integration/continuous delivery (CI/CD) pipelines are a practice focused on improving software delivery using either a DevOps or site reliability engineering (SRE) approach. 

A CI/CD pipeline introduces monitoring and automation to improve the process of application development, particularly at the integration and testing phases, as well as during delivery and deployment. Although it is possible to manually execute each of the steps of a CI/CD pipeline, the true value of CI/CD pipelines is realized through automation. A comprehensive set of application security testing (AST) tools help you test for and remediate security vulnerabilities in your CI/CD pipeline. 


SecDevOps (also known as DevSecOps and DevOpsSec) is the process of integrating secure development best practices and methodologies into development and deployment processes that DevOps makes possible. SecDevOps consists of two distinct parts:
1. Security as Code (SaC).
2. Infrastructure as Code (IaC). 

Cybercriminals take no prisoners. According to cybersecurity experts, SecDevOps, or the process of integrating secure development best practices and methodologies into the development and deployment processes, is the best way to do just that. 

Here automation comes further there are many open-source plugins and security tools are available to provide and scan our application for security tests before the development state goes to production.



A website vulnerability is a weakness or misconfiguration in a website or web application code that allows an attacker to gain some level of control of the site, and possibly the hosting server. 

A web application security scanner is a software program that performs automatic black-box testing on a web application and identifies security vulnerabilities. Scanners do not access the source code; they only perform functional testing and try to find security vulnerabilities. Various paid and free web application vulnerability scanners are available. Security Testing is a type of Software Testing that uncovers vulnerabilities of the system and determines that the data and resources of the system are protected from possible intruders. It ensures that the software system and application are free from any threats or risks that can cause a loss. 





Ruby on Rails is the most popular open-source web application framework. It’s built with the Ruby programming language. Rails help you build websites. 

Ruby on Rails is an open-source web development framework, which provides Ruby developers a time-saving alternative to develop code. It is a collection of code libraries, which offer a ready-made solution for repetitive tasks like developing tables, forms, or menus on the website. Ruby on Rails also comes with a unit testing setup called RSpec, which is very easy to learn. It uses some other scanners to scan application source code. Static analysis security vulnerability scanners are used in Ruby on Rails for scanning. So here, we can use any static analysis security tools to scan ruby on rails.


Sken.ai aligns with the security framework OWASP ZAP. It is an open-source tool which is offered by OWASP (Open Web Application Security Project), for penetration testing of your website/web application. It helps you find the security vulnerabilities in your application. Here are the OWASP top 10 security threats that your website/application might face: SQL injection, Broken authentication and session management, Cross-site scripting (XSS), Broken access control, Security misconfiguration, Sensitive data exposure, Insufficient attack protection, Cross-site request forgery (CSRF), Using components with known vulnerabilities, and Underprotected APIs. ZAP provides the following features: active scans, alerts, API authentication, and verification, etc. 

Zap is used as a proxy server, it allows the user to manipulate all of the traffic that passes through it, including traffic using HTTPS. It helps you find the security vulnerabilities in your application. Here is a list of the main features: Intercepting Proxy, Automated Scanner, Passive Scanner, Brute Force Scanner, Fuzzer, Port Scanner, Spider, Web Sockets, REST API. 

ZAP is an easy-to-use tool. Following are some more reasons for using ZAP: 

1. Ideal for both beginners and professionals 

2. Cross-platform - works across all OS (Linux, Mac, Windows) 

3. Reusable 

4. Can generate reports of the results, etc. 


A penetration test, also known as a pen test, is a simulated cyberattack against your computer system to check for exploitable vulnerabilities. In the context of web application security, penetration testing is commonly used to augment a web application firewall (WAF). Penetration testing in simple terms is a simulation of a process a hacker would use to launch an attack on a business network, attached devices, network applications, or a business website. The purpose of the simulation is to identify security issues before hackers can locate them and perform an exploit.




Mobile application security focuses on the software security posture of mobile apps on various platforms like Android, iOS, and Windows Phone. Mobile application security testing involves testing a mobile app in ways that a malicious user would try to attack it. Effective security testing begins with an understanding of the application’s business purpose and the types of data it handles. From there, a combination of static analysis, dynamic analysis, and penetration testing results in an efficient holistic assessment to find vulnerabilities that would be missed if the techniques were not used together effectively.


List of tools available in the market for all types of scanning & the sken.ai is provide same scanning features in one place:

  • DAST Scanning Tools:- 

  1. OWASP ZAP

  2. Netsparker

  3. Acunetix

  4. veracode

  5. Detectify Deep Scan

  6. SonarQube

  7. Fortify Static Code Analyzer

  8. AppScan

  9. Checkmarx

  10. Synk


  • SAST Scanning Tools:-

  1. SonarQube

  2. veracode

  3. Fortify Static Code Analyzer 

  4. AppScan

  5. Brakeman

  6. Code Climate 

  7. bundler-audit

  8. codacy

  9. VisualCodeGrepper

  10. RIPS (Re-Inforce Programming Security)

  11. Flawfinder

  12. Bandit






  • Combine Scanning (SAST/DAST/SCA) tools and alternative to sken.ai tools:-

  1. Checkmarx

  2. veracode

  3. Detectify Deep Scan

  4. AppScan

  5. SonarQube

  6. Pycharm

  7. Sucuri

  8. Pentest-Tools

  9. Netsparker

  10. OWASP ZAP 

  11. Acunetix

  12. Prospector

  13. pylint 

  14. Pyflakes

  15. Bandit

  16. Snyk

  17. Burp Suite

  18. Grabber

  19. Arachni

  20. w3af

  21. Nikto

  22. JavaScript Lint

  23. ESLint

  24. Flow

  25. JSHint

  26. Intruder

  27. Nikto2

  28. Gerrit

  29. Crucible

  30. Codacy

  31. Nessus

  32. Resharper

  33. Appknox

  34. CodeSonar

  35. Wireshark

  36. SQLmap

  37. Selenium

  38. JOHN THE RIPPER

  39. Suricata

  40. Grapl


  • Cloud Security Scanning Tools:-

  1. Rapid7 Nexpose

  2. Synk

  3. Acunetix

  4. Nessus

  5. JOHN THE RIPPER

  6. Suricata

  7. Grapl

  8. Wireshark


  • Ruby on Rails application scaling tools:-

  1. Brakeman

  2. Code Climate 

  3. bundler-audit

  4. codacy


  • SCA Scanning Tools:-

  1. OWASP Dependency Leaks.


How is it different than other tools in market:-

    I have used the sken.ai scanning tool, I have mentioned some features which I liked, 

1. There is "No App Security  Expertise" required to use this scanner.

2. "Commercial Scanners are expensive"  so mid-markets can decide to use open-source scanners. However, Open-source scanners are extremely unmanageable. There are too many of them, so it is confusing to select what you'd exactly need.

3. This includes code level scanners like SAST, SCA, Secrets, License, Test/Deploy level scanners like DAST, Containers, API, Serverless, etc. It is extremely difficult for mid-markets to operationalize scanning with these multiple types of scanners.


The mid-markets need a product that:

  1. Provides app security with no required app-security expertise. 

  2. Is affordable yet manageable.

  3. Supports comprehensive scanning across the application, yet simple to understand and use. 

Sken tackles each one of the requirements of the mid-markets. There are three pillars in Sken, and they correspond and tackle these three core requirements of the mid-markets. 

  • Sken provides a product that can be used by DevOps with no app security experience.

  • Sken packages open-source scanners in a SaaS orchestration layer and automates them in CI/CD. This SaaS orchestration does a bunch of the heavy lifting, but it makes it so simple for the end-user to operate.

  • Sken does all scan types. There includes 

  1. Code-level scanners like

    1. SAST (Static Application Security Testing) for scanning source code across multiple languages.

    2. SCA (Software Composition Analysis) for scanning open source libraries that are included in your application.

    3. Secrets for scanning open passwords

    4. License for scanning license files

  2. Test level scanners like

    1. DAST (Dynamic Application Security Testing)

  3. Deployment level scanners

    1. Containers

    2. API, etc.

It provides free scanning forever for single users, Unlimited tests on open source projects, 300 tests per month across all apps on private projects, Priority Scoring, and Continuous Monitoring. As it is a freemium (with about 300 free scans per month) it is the best choice for beginners and small project developers.


Why should you use Sken.ai:- Because of the number of different features sken.ai provides, It helps in static code analysis, Source code analysis, code review, code complexity, and penetration testing free of cost. As it is a freemium (with about 300 free scans per month) it is the best choice for beginners and small project developers.

What are the integrations possible in sken.ai:- Sken.ai integrates with the CICD pipeline with 3 simple steps. 1. Add Sken.ai in CICD. 2. Write sken.yml as you need. 3. Review issues in the Sken.ai portal. It uses automated open-source scanners providing tools like SCA, SAST, DAST, Secrets management, and many more. Those are included:-

  1. SCA: OWASP Dependency Leaks.

  2. SAST: Find-Sec-Bugs, NodeJScan, Brakeman, Bandit, Gosec, PHPCodeSniffer, ES Lint, TSLint.

  3. DAST: OWASP ZAP

  4. Secrets: Gitleaks, Trufflehog.


Here is the demo of sken integration with jenkins and travis-ci:-

  1. Jenkins integrate with sken:- https://drive.google.com/file/d/1OxpziWQZ6lEZ4Q6c7WYWAw3DTNf9qLd6/view?usp=sharing

  2. Travis CI integrate with sken:-

https://drive.google.com/file/d/1-zEbJ0rwzkR_vpvDkBvTXY4MwRXZcOFX/view?usp=sharing



Saturday, 9 March 2019

DevSecOps Pune Meetup 4



9th March 2019

This is our 4th DevSecOps meetup. More and More and More swag sponsors added to the list. New sponsors: Elastic, Sonatype, Synk along with previous sponsors i.e. Polyverse and Cloudneeti already into the list. The meetup head count was usual and exact. We have been hitting the correct and intended audience.

Qualys Pune was the venue for the meetup. This is the 2nd time we organized the meetup at Qualys. We started at 10.30 am. We had a huge list of topics this time and I was pretty sure not all of them are going to get discussed since the most voted topics were very interesting. The following topics got discussed:

  • Machine Learning for Security: This topic got discussed for a good 40 minutes although none of us was an ML engineer and only understood theoretical ML concepts. The topic was chosen more for a discussion as the person suggester of the topic just had the curiosity to see if this can be done. The topic unfolded as people discussed and understood how ML works and how is it being used in the industry. Different tools like TensorFlow, Pandas, etc got discussed. Once we had a clear picture of ML, we moved to understand what security breaches we experienced in our Ops lives in industry. Based on our experience, we framed logic as to how ML algorithm could be written by Data Science engineers by studying security auth logs and application logs. We also discussed how ELK stack could be used to prove a security attack on system and further. We agreed to the part that as an Ops person we can only best provide inputs and prove security issues in systems. An ML engineer should be the best person to provide inputs on what ML algorithms could be used to mitigate security issues.
  •  Ansible for DevSecOps CI/CD pipeline: Ansible was suggested multiple times in the past meetups as well but always went not noticed and never got discussed. In this session, we started with Ansible and went on an on for a good 1 hour. We understood how Ansible is best used in different organizations. We also discussed how a bad code in Ansible can mess up and not make any difference between olden days shell script. We discussed Idempotency feature of Ansible. We also discussed as to how Ansible is being used by some firms for Provisioning, Configuration Management and Deployment altogether. We agreed how many used Puppet for multiple jobs and later moving to a different tool was difficult for them. Hence using different tools for specific different jobs could be useful. We discussed a typical CI/CD workflow with Terraform for provisioning, Ansible for CM and Fabric/Capistrano or different language specific deployment tools for easy deployments and rollbacks. We also did a small white board presentation on how CI/CD can be used for all 3 purposes. We also discussed DR strategies, Cloud managed services like auto-scaling services etc.


  • Git for securing code : The suggester was a Developer and knew exactly how was not being used in the best manner to ensure security and highlighted some best ways like code review practices, git hooks to be used in Jenkins, linting analysis using pre-commit hooks, static code analysis with SonarQube before merge. We discussed this in details already in CI/CD pipeline above as well.
  • Burpsuite :  Burpsuite was a tool that many knew but never got a chance to use as such. The suggester of the topic had extensive experience using it and spoke about multiple possibilities and usecases that could be achieved using Burpsuite. He also highlighted on how precautions need to be  taken before using this tool for testing giving examples of one of his projects. Burpsuite indeed was very interesting for many members and we decided to have a demo session on this.
Takeaways from this session for speakers to prepare were:
  •  Session required on ML in security
  • Demo on CI/CD in DevSecOps
  • Demo on Burpsuite
Many topics that did not get discussed were:
  • Nexus scan in Jenkins pipeline
  • SonarQube for secure code analysis
  • Pentesting with Python
  • IOT Security
  • Metasploit for pentesting
  • Cloud security
  • Prowler
  • Automobile software release cycle and missing lacking devops chain
  • Maven
  • Regulations
In the end Shirish wanted to highlight the recent All Intel Chips Open To New 'Spoiler' Non-Spectre Attack

Some clicks :)














Wednesday, 20 February 2019

Source build Envoy proxy on Ubuntu 18.04

sudo apt-get update
sudo apt-get install openjdk-8-jdk build-essential autoconf libtool cmake ninja-build
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install bazel

wget https://dl.google.com/go/go1.11.5.linux-amd64.tar.gz
tar -xvf go1.11.5.linux-amd64.tar.gz
sudo chown -R root:root ./go
sudo mv go /usr/local
echo "export GOPATH=$HOME/go" >> ~/.profile
echo "export PATH=$PATH:/usr/local/go/bin:$GOPATH/bin" >> ~/.profile

 git clone https://github.com/envoyproxy/envoy.git
cd envoy/
bazel build --package_path %workspace%:/home/<user>/envoy/ //source/exe:envoy-static

RELAX, ITS GOING TO TAKE LONG TIME




To generate the example configurations run the following from the root of the repo:

mkdir -p generated/configs
bazel build //configs:example_configs


RELAX, ITS GOING TO TAKE LONG TIME

I wrote an Ansible playbook too for this. I will publish it later on Github

Easily SCP/Rsync through bastion host or SCP/Rsync through multiple hops

Often we work in environment where we need to copy files or directories from a local system to another server that can be accessed only through a Bastion host. In such cases, typically we transfer from local machine to Bastion and from Bastion to the intended server. This is time consuming, repetitive and unreliable too. There are many ways you can make this automated. I found a way to get this done through SSH tunneling. Here's how it works:

There are 3 machines involved here:
  1. localhost
  2. Bastion host
  3. Intended server

1. Create an SSH tunnel from localhost to the intended host through bastion. The tunnel will be created from port 1234 at localhost. You may choose any other port.
ssh -L 1234:<intended_server>:22 <user>@<bastion-host> cat -
2. In a new tab initiate the file/directory transfer using the tunnel port
scp -P 1234 <file_to_transfer> <user_of_intended_server>@127.0.0.1:~/

As I did this, I realized SCP is very slow in getting the transfer done due to its linear and sequential file transfer behavior. Hence, I used Rsync which made it pretty fast due to its delta based transfer algorithm

rsync -avz -e "ssh -p 1234" <file_to_transfer> <user_of_intended_server>@127.0.0.1:~/

Saturday, 9 February 2019

DevSecOps Pune Meetup 3



9th Feb 2019

This is our 3rd DevSecOps meetup. I am glad to see the head count getting even better. Another Goodies sponsor added to the list. Cloudneeti Software sponsored T-shirt and Mugs to best presenter.

For this meetup, we got an exact count again with 2-3 last minute exits. Cloudneeti Software was the venue for this meetup. We started sharp at 10.30. Surprisingly the topics discussed this time were pretty advanced and much looked out for. The following topics got discussed:
  •  Kali Linux for security : The topic was super hit and got discussed for pretty long hours. We spoke for a good 20 minutes on Kali Linux and only stopped because the next topics were also equally interesting. We spoke about the old distro Backtrack and how Kali got introduced. Others added their inputs as to how Kali is used by different SecOps people in industry. Overall we all agreed that Kali was a really vast topic to be just discussed and this rather needed a presentation + demo with so many tools within.
  • OWASP Top 10 web app security risks: As we discussed Kali, we also spoke about OWASP top 10 in the same discussion and this got prolong for another 20 mins. Not all of us were aware of all OWASP top 10 attacks hence we Googled it just for making notes, however with time limitations we could only discuss the most famous SQL Injection and a little about XSS. We also concluded that this needed a bigger session.
  • CI/CD in DevOps Pipeline: DevSecOps CI/CD pipeline was discussed shortly in 1st meetup but we did not deep dive in it then. In this meetup we actually went to the depth of Jenkins, Git, SonarQube, Static code analysis, Container Image security, SecOps role to play, Vulnerable libraries being used by devs and how to resolve this. How Github has inbuilt vulnerability analysis done now and many more details. We also drew the following CI/CD architecture diagram for DevSecOps pipeline.
  • Ansible secure key rotation: This was more of a question to the forum as to how this could be done since Chef and Puppet use clients and use their own keys for security, while Ansible uses SSH. We agreed to the fact that the keys must get rotated and many companies do follow this. Ansible's authorized_keys module helps you to rotate keys was the perfect answer suggested.
  • Security Patching at scale in cloud: This was again a question and the answer was many companies did this using cloud native tools to create a golden images and other tools to get into the image to verify if the images were CIS compliant this is not just for images but also for Cloud environment and resources in general. Once again Prowler was discussed to evaluate the cloud environment.
Takeaways from this session for speakers to prepare were:

  •  Kali Linux hands on demo
  • OWASP Top 10 risks
  • CI/CD in DevSecOps
A few topics that did not get discussed were:
  • Metasploit for Pentesting
  • GDPR automation
  • Deploying Software securely
  • Cloud security trends 2018-19
An important Topic that Budhram did put forth as to what minimum qualifications and expertise do companies look for in a fresher candidate to call him an eligible candidate for DevOps Engineering further. This was a good debate cum discussion that we all spoke about in the end.

This turned out to be a long event in spite of small number of attendees. Ashish, Budhram, Shrikant and Dhiru got goodies to keep discussion more happening and actively participating in all topics as well. Some others got stickers.

Some clicks :)







Copying ssh keys easily

I use VMs/Vagrant a lot in my day work for all sysadmin/devops automation. One of the problems that I always face with the systems is to authorize my server for the 1st time with my master host. If I am using 10 VMs I need to authorize them 10 times? I wrote a small script to automate this process:

  1. Create a "list" file and add all IPs and hostnames for the VMs in it.
  2. Create a "password" file to write your SSH password in it, you may choose to write the password in the bash, however I feel this gives me the flexibility to add the bash to my source code if need by putting password file in a.gitignore
  3. Next create a shell script that will read the IP addresses and the hostnames from the "list" file and password/s from the password file(I generally keep the same password for all VMs for simplicity)
  4. Remember 2 commands are useful here ssh-copy-ip and ssh-keyscan. Here's how you use them:
 
    ssh-keyscan -H <IP> >> ~/.ssh/known_hosts                                                                                                                                
    sshpass -f <password> ssh-copy-id -i ~/.ssh/id_rsa.pub <USER>@<IP>                                                                                                      
The ssh-keyscan command command is for gathering the public ssh host key of a VM host specified. After collecting the publich ssh-key it adds it to your localhost. You can verify this by checking the contents of "~/.ssh/known_hosts"

The ssh-copy-id command copies the public key of your default identity (otherwise use -i identity_file for other identities) to the remote host. You can verify this by checking the content on ~/.ssh/authorized_keys in the VM host.

The final script looks like this with a loop:

#!/bin/bash                                                                                                                                                                 
user="vagrant"                                                                                                                                                              
for ip in `cat ./list`; do                                                                                                                                                  
    ssh-keyscan -H $ip >> ~/.ssh/known_hosts                                                                                                                                
    sshpass -f password.txt ssh-copy-id -i ~/.ssh/id_rsa.pub $user@$ip                                                                                                      
done 


That's how my "list" file looks like:



consul-server1                                                                                                                                                              
consul-server2                                                                                                                                                              
bootstrap-server1                                                                                                                                                           
client1                                                                                                                                                                     
client2                                                                                                                                                                     
client3                                                                                                                                                                     
client4                                                                                                                                                                     
client5                                                                                                                                                                     
192.168.3.111                                                                                                                                                               
192.168.3.112                                                                                                                                                               
192.168.3.121                                                                                                                                                               
192.168.3.151                                                                                                                                                               
192.168.3.152                                                                                                                                                               
192.168.3.153                                                                                                                                                               
192.168.3.154                                                                                                                                                               
192.168.3.155                                                                                                                                                               

I purposely add IP as well as hostname as I keep using them interchangeably. I also came to know about ansible authorized_keys module that does the ssh-copy-id task:
- name: Set authorized key for user ubuntu copying it from current user
  authorized_key:
    user: ubuntu
    state: present
    key: "{{ lookup('file', lookup('env','HOME') + '/.ssh/id_rsa.pub') }}"
However, you will still need the the ssh-keyscan here. This script goes handy for ops who keep destroying their local environment and use a new one.This is available on Github: https://github.com/iamrawtion/ansible-autossh