Episode #13 - Artifact Repositories

Last episode recap

• In the last episode, we discussed the Sarbanes-Oxley law and how it impacts publicly traded companies, what regulatory controls that have to be in place for a company, and what the impact is for non-compliance or fraud.

Summary of this episode

• In this episode, we are going to discuss where you should store your compiled or “built” artifacts. These tools are referred to as binary repositories or artifact repositories. We will discuss the types of repos that you can create, what artifact types you can store and also the security features for each of the tools.

Episode Content

What’s in it for you?

• After listening to this episode, my hope is that you will have a better understanding of what a binary repository is and what it is used for as well as some of the capabilities of 2 of the offerings in the industry.

What is a binary repository?

• A binary repository is simply a place to store your compiled or “built” artifacts. Some people have in the past used shared network drives or a folder on their laptop. These are not optimal solutions since you can accidentally delete single items or the whole folder itself.

What can you put in it?

• There is no limit to the items that can be uploaded to a binary repository. Simple rule of thumb, no un-compiled or un-packaged source code should be uploaded to a binary repo.

What are the types of repositories that you can use?

• Release – release-able artifacts – ready for installation into an environment
• snapshot – used for builds that are done on a set schedule (daily, hourly, etc.)
• dependency – used to hold artifacts that your build will need to complete successfully but may not be readily available from another source.

What are the major binary repositories?

• Sonatype Nexus
• Jfrog Artifactory

What are remote proxies in a binary repository?

• Remote proxies are typically sites hosted internally to your organization that points to an external site. This is usually done to enhance security to sites that you cannot verify that the content has been properly secured. Here are some examples:

• Maven central
• npm
• docker hub
• etc…

What security tools come with the binary repository?

• Nexus Firewall – add-on product (has a cost)
• Jfrog Xray – add-on product (has a cost)

What are some of the similarities of the 2 vendor tools?

• The both allow you to upload various types of artifacts.
• They both have download mechanisms for existing artifacts.
• They both support release and snapshot repos.
• They both can maintain a dependency repo for internal artifacts needed by builds.
• They both support versioning and tagging of artifacts.

What are some of the differences?

• They both store the artifacts in a different way
• They both store metadata in a slightly different method.

What are the pro’s of each tool?

• Nexus has a built in capability around Maven Central since that company maintains Maven Central itself.
• In my experience, Jfrog Artifactory was the first to support all the different flavors of remote proxies, but both currently support the majority.

What are the con’s of each tool?

• Confusing documentation on both parts. I have experienced random documentation updates in the middle of an installation that required that I re-do part of the work I had just completed.

Recap of this episode

• In this episode, we discussed what binary repositories are, what you can store in them, what the similarities are between the products, and some of the differences.

Next Episode: Wyze home automation hardware review

Episode #12 - Ball and Chain of Custody - Legal Requirements

Last Episode recap

• In the last episode we discussed Jenkins. We talked about what it is, what it is not and how it can be configured to use in your automation efforts.

Summary of this episode

• I hope you have seen that I’m trying to do these episodes in what I think is a very specific order. We will build on each episode so that you understand how each topic discussed can be used to enhance your automation effort.
• In this episode, we will discuss maintaining a software “chain of custody”, why it is important, what happens if you don’t and how your automation can help you do it.

What's in it for you

• A better understanding of the importance of maintaining good software controls and how it keeps you and your company out of trouble.

Episode Content:

• Why does it matter?
  • In 2001, a large corporation named Enron was embroiled in a controversy involving its financial records. It was followed by other controversies with other large publicly traded corporations.
  • In July of 2002, two U.S. Congressmen, Sen. Paul S. Sarbanes and Sen. Michael G. Oxley, sponsored a bill to rein in the fraudulent financial reporting that led to these massive corruption scandals. This bill was titled the Sarbanes-Oxley Corporate Responsibility Act of 2002 (or SOX for short).
  • The bill tightened controls around how a company recorded and reported its financial Information.
  • One of its goals was to provide for structured, consistent and repeatable controls.
  • It was created as a way to for the leaders and employees of a company to create and maintain a culture of corporate financial responsibility.
  • It provided for the officers of a company to “sign” their financial statements. This means that they take personal responsibility that they are accurate and correct with no malfeasance or attempts at fraud against investors.
  • It increased the fines and sentences for individuals that attempt to commit fraud or illegal financial practices.
  • It requires a company to document its internal control processes.
  • It requires a company to to have an external auditor to certify that their internal control processes are being followed and that the financials are accurate.
  • It dictates that all off-balance sheet transactions must be reported.
  • It provides the Security and Exchange Commission more power to perform investigate any company suspected of violations.

• Who has to comply?
  • All publicly traded companies
  • All subsidiaries of foreign companies in the U.S.
  • All private companies that are preparing for an IPO
• Fines and Prison Time
  • Non-compliance could result in fines up to $1 million dollars and up to 10 years in jail
  • If a company intentionally defrauds investors, it could result in $5 million dollars in fines and up to 20 years in jail.

• When should you worry about it?
  • You shouldn’t worry about it too much but,
  • You should make sure that your control processes are well documented, your staff understands why they are in place, and any consequences that they or the company might face if they do not follow the controls.
• Where should they be implemented in your SDLC?
  • In my opinion, every step of the SDLC, for example:
    • Requirements – should be documented and changes should be tracked.
    • Development – All source code should be in an SCM system so that any changes can be tracked back to why and who made the change.
    • Build – All builds should be documented and all artifacts of the build process should be stored and versioned.
    • Testing – all test cases that are used to verify requirement functionality should be documented and any results of those test cases related to a specific build should be tagged with the build ID and archived after completion.
    • Environments – All changes to an environment used for a specific build should be tagged and documented for approval prior to implementation.
    • Deploy – The reason for the deployment, date and time, and approval should all be documented.
    • Promotion to a higher environment – all approvals should be documented and tracked for each environment promotion.

• How can your automation help you with all this documentation?
  • By using some or all of the tools that we are discussing, you can have a lot of this documentation done for you automatically by your automation.
  • For Example:
    • Requirements: Jira, Quality Center
    • Development: Github, Bitbucket, SVN, Team Foundation Server
    • Build: Maven, Gradle, Jenkins, CI/CD pipelines
    • Testing: qTest, Quality Center, Bugzilla
    • Environments: CI/CD, Jira
    • Deploy: CI/CD, Jenkins, Ansible, Terraform
    • Promotion: Jira, Remedy, ServiceNow

• Each of these tools will automatically record the changes that you make, builds that run, tests that are performed, and artifacts that are deployed.
• Using Jenkins or a similar tool, you can create a pipeline or job that can automatically create the files required by your audit partners to run their reviews.
• Usually the only changes to be reviewed after automation are any changes that are made to the automation pipeline/jobs themselves.
• Working with internal and external auditors can be a daunting task since they are trained to ask many probing questions to identify problems with your process.
• Using automation makes their and your job easier when it is time to perform these control reviews, which is usually monthly, quarterly, semi-annually or annually.

recap of this episode

• In this episode we discussed what software chain of custody is, why it is important, where it originated, how it is controlled, and how your automation can assist you in financial reporting compliance audits.

Next Episode: Artifact repositories

Episode #11 - Tools - Jenkins - Not your everyday butler

Last Episode recap

• In the last episode, we discussed CI/CD and related build topics.  We talked about what good CI should look like and what components make up that CI.  We talked about CD and how that is typically done and also dis-jointed or dis-associated CI and CD.

Summary of this episode

• In this episode, we are going to discuss Jenkins and how it can be used in your automation.

What's in it for you

• Once you listen to this episode you should have a better understanding of what Jenkins is and how it can be used, the types of tasks that can be automated, and some of the items that can be configured (plugins) to enable automation.

Jenkins description

• what is it
  • Jenkins is an automation platform that can perform a variety of tasks to enable your automation.  It can be used to run your CI/CD  pipelines integration your SCM to your automation all the way to your production environment.
• what's it used for
  • build jobs
  • testing jobs
  • deployment jobs
  • reporting jobs
  • auditing jobs
  • monitoring jobs
• good use cases
• types of jobs
  • freestyle
  • pipelines
  • multi-branch pipeline
• plugins
  • examples:  folders, pipelines ( many different plugins ), git, ansible, artifactory, azure app service, credentials, docker, generic webhook, groovy, mask passwords, nexus artifact uploader, SSH plugin, and many more.
  • when are they useful and when should you use them
    • usually when a vendor has an integration plugin and you can’t put it into your pipeline as code
    • or when it is a very specialized piece of functionality that you cannot easily code
  • when should you not use them
    • when you can easily code the functionality in your pipeline code (ssh to another server perform a task)
• agent nodes
  • accessing agent nodes
  • ssh creds
• folder security
  • matrix based security
  • LDAP security
• Integrations
  • SCM
  • others

recap of this episode

• Description
• Use cases
• Job types
• Plugins
• Agent Nodes
• Security
• Integrations

Episode #7 - Tools - Github

Description of GitHub
- hosted platform for managing your source code from creation/ideation to production
- built-in actions and workflows to control your code build and deploys while not leaving GitHub's platform.

GitHub Project Structure
- It can be as varied as the languages that you can code in.

readme files
- how to tell the world about your GitHub project
- uses markdown for formatting

Pros/Cons of using GitHub
- Easy access for anyone with an internet connection
- Built-in process to enhance automation
- Its in the cloud....always a risk
- Only as good as what you put into it
- Concerns about Intellectual Property
- Concerns about security tokens, passwords, IDs, SSH keys, etc.

Workflows
- automate tasks directly in GitHub

Webhooks
- Used to drive automation tasks with other systems (or within GitHub itself)

GitHub Pages
- a website for your project....done right from your GitHub repo
- can use a custom domain name that you own

Integrations
- Integrates with almost any tool via Webhooks
- GitHUb has developed a lot of their own offerings around certain technologies

Code Review
- It all starts with a pull request
- diffs, history, blame
- embedded chat
- conflict resolution
- Integrations with other code review tools

GitHub Actions (or features/workflows)
- automation workflows built into GitHub
- run on any event
- hosted runners for most major OS flavors
- parallel builds for different OS or browser combination
- supports many languages
- users can build their own runners
- live logging for real-time build debugging
- built in secrets management

GitHub Products
- Atom - IDE
- Electron - Open source cross platform development with javascript, html, and css
- GitHub LFS - Large file storage offering
- IDE Plugins
- Mobile App
- GitHub Marketplace
- GitHub Security Lab

Git Merge Conference in March 2020 - Sponsored by GitHub

Coming up next - A look at BitBucket - Atlassian's Git offering.

Episode #5 - Build Tech and Value Stream

Episode 5 - Build Tech and Value Stream
---------------------------------------
So you have your code, what do you do with it?
- Bad idea - keep it on your laptop or a network drive

What SCM tool do you use to store your source code?
- SCM - Software Configuration Management
- Also called version control
- It's where you put your code so that nothing bad happens to it (like being deleted).

What build tool you use will depend on the platform and language that you are writing your code in.
- Java, .net, C++, C#, nodeJS, etc.
- Linux, AIX, Solaris, Windows, Docker, Mainframe, etc.

First you will need to understand your build process.
- Automation enables your process to go faster and smoother but it does not replace the process.
- You need to sit down and document the entire process from Requirements definition to having an artifact that can be installed into an environment.
- Value Stream Mapping
-example:
- Document the requirement (including acceptance criteria).
- Check out code (if existing).
- Modify code to meet the acceptance requirements as written in the requirement.
- Check in / Commit your code to your SCM system.
- Update and parameter or config files as needed.
- Run automated/manual unit tests (this will be another podcast).
- Run your packaging utility (this will depend on the language that you are writing your code in).
- Ant, Maven, Gradle, Make, etc.
- Upload Artifact to server (host).
This step will be replaced when automation is written.  The artifact will be uploaded to an artifact repository (binary repository).
- Install artifact into testing environment on this host.