Gremlin Chaos Workflow - Breaking Things on Purpose

Gremlin Chaos Workflow - Breaking Things on Purpose

Chaos Engineering is an up and coming space where injecting failure helps build system resilience. Your CD Pipeline is an excellent place to experiment with Chaos Engineering. Working with our friends at Gremlin, a leader in Chaos Engineering, a simple integration flow between Harness and Gremlin.

The first step is to sign up for Harness and Gremlin accounts if you have not already done so.

Gremlin UI:

https://app.gremlin.com

Make sure to take note of your Gremlin Keys for API access.

ID Key
Team ID https://app.gremlin.com/settings/teams
Secret Key In Account
Bearer Token CURL below

Make sure to get your Bearer Token from Gremlin which will be needed for API Calls.


curl -X POST --header 'Content-Type: application/x-www-form-urlencoded' \

--data-urlencode 'email=YourEMAIL' \

--data-urlencode 'password=YourPW' \

'https://api.gremlin.com/v1/users/auth?getCompanySession=true'

Gremlin has an awesome blog post about leveraging your first Kubernetes Attack. Attacks are the Chaos Engineering tests that Gremlin orchestrates.

I like to leverage Amazon EKS and Weave’s EKSCTL to create a cluster. Heads up make sure to keep EKSCTL updated since changes in EKS can impact how the cluster is started.


#Create EKS Cluster

eksctl create cluster \

--name gremlinchaos \

--version 1.15 \

--region us-east-1 \

--nodegroup-name standard-workers \

--node-type t3.xlarge \

--nodes 2 \

--nodes-min 1 \

--nodes-max 3 \

--node-ami auto

Harness UI:

https://app.harness.io

The easiest way to try out Gremlin is to wire up a Harness Kubernetes Delegate.

Installing a Harness Delegate in a Kubernetes is pretty simple.

Setup->Harness Delegates -> Download Delegates -> Kubernetes YAML

Download and can give a name like ”gremlin” [which you can see I ran through already :-)]. Unzip the TAR and install with kubectl.

kubectl apply -f harness-delegate.yaml

Once up, you will see the Delegate listed.

With the Delegate Installed, next can add the Kubernetes Cluster to Harness.

Setup -> Cloud Providers + Add Cloud Provider -> Kubernetes Cluster. Can inherit the cluster details from the Delegate.

Once you click submit, your K8s cluster will be available.

The next step is to wire up the Gremlin Helm repository to Harness. Can take a look at the Helm Chart Values in GitHub in case any more modifications are needed above and beyond our example.

Setup -> Connectors -> Artifact Servers + Add Artifact Server

Repository URL: https://helm.gremlin.com

The next step will be to create a Harness Application to house your Workflow.

Setup -> Applications + Add Application

Next, add a Service which will be a Kubernetes based deployment.

Setup -> Gremlin Chaos App -> Services

With the Service Created, link the Helm Manifest as a Remote Manifest by clicking on the ellipses on the side. The chart name is “gremlin”.

From the Helm Chart, you will need to provide the Helm Chart certain Values for your account. This can be done in the Configuration section in Values YAML Override. Also, you’ll need your Kubernetes cluster name [ours is gremlinchaos]

Add Inline Values

The Values


gremlin:

secret:

teamID: YourTeamID

teamSecret: YourTeamSecret

managed: true

clusterID: gremlinchaos

With the Helm values wired, let’s create a Harness Workflow to lay down Gremlin and run your first Attack.

First will need to wire the Kubernetes Cluster as a Harness Environment.

Setup -> Gremlin Chaos App - > Environments + Add Environment

Don’t worry about Environment Type, that is for label purposes.

Next, add an Infrastructure Definition. Will be a Kubernetes Deployment type. You can change your namespace to gremlin per Gremlin’s example if needed.

Add a Workflow.

Setup -> Gremlin Chaos App -> Workflows +Add Workflow

In the Workflow, below the Rolling Deployment is a Verify Step. Let’s add a step called “Call Gremlin API”.

Gremlin has a great list of API Call Examples to help.

Add Step -> New Step -> Utility -> Shell Script

The Shell Script has this format:


curl -X POST \

--header "Content-Type: application/json" \

--header "Authorization: Bearer $yourToken" \

https://api.gremlin.com/v1/attacks/new?teamId=$yourTeamID \

--data '

{

"command": { "type": "cpu", "args": ["-c", "1", "--length", "30"] },

"target": { "type": "Random" }

}'

With all the wirings up, time to run this bad boy!

Click on the Deploy icon in the Workflow Overview or can launch from the Continous Deployment UI.

Watch the Succes in the Harness UI:

Can head back to the Gremlin UI and see the Attack in progress.

https://app.gremlin.com/attacks/infrastructure

And just like that, you just ran your very first Gremlin Attack from Harness!

Art of the possible we are working on integrating with our Continous Verification and moving all the secrets to a Harness Secret manager which will end up on the Harness Blog.

Cheers!

-Ravi