Lab Environment

Topology

This lab environment consists of the following components:

  • AWS Cloud9 (Browser based IDE backed by EC2)
  • TeaStore – The TeaStore is a reference and test micro-service application which emulates a basic web store for tea and tea supplies.
  • On-Premises Infrastructure
    • Cisco Intersight Kubernetes Service (IKS)
    • Cisco HyperFlex running VMware ESXi 6.7
  • Public Cloud: AWS – Amazon Elastic Kubernetes Service (EKS)
  • Payment Gateway running externally
  • AppDynamics
  • ThousandEyes
  • Intersight

The diagram below provides illustrates the deployment of the TeaStore lab application.

In this section we’ll begin with some prerequisite tasks you’ll need to complete to setup the lab environment before deploying any observability agents.

Lab Setup

Access Your Cloud9 Instance

Your instructor will provide you with credentials to log into the AWS console and the link to the Cloud9 instance you will use during the labs.

🔵 Use the URL your instructor provided to get to the login page of the AWS console.

🔵 Use the Account ID, the IAM user name, and Password your instructor provided to login.

🔵 When you login you will be taken directly to your Cloud9 instance.

🔵 Open a new terminal window in your Cloud9 UI.

Validate Environment Variables

🔵 Use the command below to validate the lab number variable is set to your assigned lab number.

echo $fso_lab_number

The output should look like the image below.

🔵 Use the command below to validate the AWS Region variable is set.

echo $aws_region_name

The output should look like the image below.

🔵 Use the command below to validate the IKS cluster name variable has your assigned lab number in the name.

echo $iks_cluster_name

The output should look like the image below.

🔵 Use the command below to validate the EKS cluster name variable has your assigned lab number in the name.

echo $aws_eks_cluster_name

The output should look like the image below.

🔵 Use the command below to validate the IKS Kubeconfig file variable has your assigned lab number in the file name.

echo $iks_kubeconfig_filepath

The output should look like the image below.

Login to Intersight

🔵 Log into Intersight

🔵 In the lab, OKTA is used to access both Intersight and ThousandEyes. The instructions for logging into OKTA are located at this link: https://fso.cisco-one.com/okta/

Obtain IKS Kubeconfig

🔵 Once you are logged in, from the drop-down menu, select Infrastructure Services.

🔵 Use the steps below to download the IKS Kubeconfig file to your local file system.

  1. Click on the ‘Operate’ tab on the left menu
  2. Click on the ‘Kubernetes’ tab
  3. Type in {cluster-prefix}-FSO-Lab-{your-lab-number} in the search box and hit enter
  4. Click on the three dots on the far right of you IKS Kubernetes cluster
  5. Select the ‘Download Kubeconfig’ drop-down option
  6. Save the file to your local file systemU

Now upload the IKS Kubeconfig file to your Cloud9 file system using the steps below.

On the left hand file system tree select the root directory that is ’/home/ec2-user’
From the top menu select ‘File > Upload Local Files …’ then click “Select files” and then find and select the IKS Kubeconfig file on your local file system to upload it

🔵 Use the commands below to validate the IKS Kubeconfig file is on the Cloud9 file system

cd /home/ec2-user

ls -l

The output should look like the image below.

🔵 Use the command below to view the worker nodes in the IKS Kubernetes cluster.

kubectl get nodes -o wide –kubeconfig $iks_kubeconfig_filepath

The output should look similar to the image below listing your cluster.

Obtain EKS Kubeconfig

🔵 Use the commands below to pull down the EKS Kubeconfig file to your Cloud9 instance.

cd /home/ec2-user

aws eks –region $aws_region_name update-kubeconfig –name $aws_eks_cluster_name

The output should look like the image below.

🔵 Use the command below to view the worker nodes in the EKS Kubernetes cluster.

kubectl get nodes -o wide

The output should look similar to the image below.

Deploy TeaStore Application

🔵 Use the commands below to deploy the TeaStore application to the IKS and EKS Kubernetes clusters.

cd /opt/fso-lab-devops/provisioners/scripts/common

./deploy_teastore_application_to_hybrid_kubernetes.sh

The output should look like the image below and the deployment should take ~8 minutes to complete.

Confirm TeaStore Deployment

🔵 Use the command below to confirm deployment to the IKS Kubernetes cluster.

$aws_region_name

kubectl get pods -o wide –kubeconfig $iks_kubeconfig_filepath

The output should look like the image below.

🔵 Use the command below to confirm deployment to the EKS Kubernetes cluster.

kubectl get pods -o wide

The output should look like the image below.

🔵 Use the command below to obtain the URL to the TeaStore UI.

teastoreurl

The output should look like the image below.

🔵 Now left-click on the URL and select ‘Open’.

🔵 Now click on the ‘Status’ link to confirm the status of all the services.

Run Workshop Setup

🔵 Use the commands below to run the workshop setup utility.

cd /home/ec2-user/environment/workshop

./setupWorkshop.sh

The output should look like the image below.

What steps are being performed with this setup utility?

  • Populates the AppDynamics Controller connection settings for the AppDynamics agents
  • Deploys the AppDynamics Database agent to the IKS Kubernetes cluster
  • Creates the AppDynamics Database collector in the Controller allowing the Database agent to monitor the database
  • Creates various other artifacts in the AppDynamics Controller (APM Application, RBAC User & Role, License Rule, Custom Dashboard)
  • Retrieves the Intersight API Key used for the IWO Action Integration (IWO Widget on Custom Dashboard)

Deploy Metrics Server

🔵 Use the commands below to deploy the metric server to both the IKS and EKS cluster.

cd /home/ec2-user

 kubectl apply -f ./metrics-server/metrics-server-eks.yaml 

kubectl apply -f ./metrics-server/metrics-server-iks.yaml –kubeconfig $iks_kubeconfig_filepath

The output should look like the image below.

Now that you’ve completed all the setup prerequisites for your lab environment and deployed the TeaStore application, it’s time to deploy the different observability agents.

DEPLOY AGENTS

🔵 Use the commands below to deploy the AppDynamics agents to the IKS and EKS Kubernetes clusters

cd /home/ec2-user/environment/workshop

./deploy_appdynamics_agents.sh

The output should look like the image below.

🔵 What AppDynamics agents were deployed and how?

Agents Deployed:

  • Cluster Agent
  • Java APM Agent
  • Server Monitoring Agent
  • Network Visibility Agent
  • Secure Application Agent

Though there are several different ways to deploy these agents, we’ve used the AppDynamics Helm Chart that simplified deployment to the Kubernetes clusters.

Deploy IWO Collectors to EKS

🔵 Use a terminal window in your Cloud9 instance to perform the next series of steps.

🔵 Use the commands below to create the namespace in the EKS cluster for the IWO collectors.

cd /home/ec2-user/iwok8scollector

kubectl create namespace iwo-collector

The output should look like the image below.

🔵 Use the commands below to deploy the IWO collectors to the EKS cluster.

cd /home/ec2-user

helm install –debug my-iwo-k8s-collector ./iwok8scollector/ –namespace iwo-collector –set iwoServerVersion=8.3 –set collectorImage.tag=8.3.1 –set targetName=$aws_eks_cluster_name

The output should look like the image below.

Deploy IWO Collectors to IKS (Reference Only)

While the steps to deploy the IWO Collectors to the IKS cluster are defined below, these steps have already been done for you in this workshop. The steps are here for your reference so you have all the steps documented that were part of the complete IWO setup. Do not do these steps.

🔵 Use the commands below to create the namespace in the IKS cluster for the IWO collectors.

cd /home/ec2-user/iwok8scollector

kubectl create namespace iwo-collector –kubeconfig $iks_kubeconfig_filepath

The output should look like the image below.

🔵 Use the commands below to deploy the IWO collectors to the IKS cluster.

cd /home/ec2-user

helm install –debug my-iwo-k8s-collector ./iwok8scollector/ –namespace iwo-collector –set iwoServerVersion=8.3 –set collectorImage.tag=8.3.1 –set targetName=$iks_cluster_name –kubeconfig $iks_kubeconfig_filepath

The output should look like the image below.

Claim Target for EKS

🔵 Use the commands below to verify the deployment of the IWO collectors in the EKS cluster.

kubectl get pods -o wide -n iwo-collector

🔵 Highlight and copy the name of the first pod in the list of pods returned for the EKS cluster and save that pod name for the next step.

🔵 Use the command below to access the IWO collector running in EKS. Use the collector pod name running in EKS that you obtained previously to substitute {your-eks-pod-name-saved-previously} in the command below.

kubectl -n iwo-collector port-forward {your-eks-pod-name-saved-previously} 9110

The output should look like the image below.

🔵 Open a new terminal window in Cloud9 and use the command below to return the Device ID for the IWO collector running in EKS.

curl -s http://localhost:9110/DeviceIdentifiers

🔵 Now highlight the Device ID (without including the double quotes) and save that ID in a text file.

🔵 Next use the command below to return the Security Token for the IWO collector running in EKS.

curl -s http://localhost:9110/SecurityTokens

🔵 Now highlight the Security Token value (without including the double quotes) and save that value in a text file.

🔵 Use the credentials for Cisco Intersight via OKTA provided by your instructor to login to Intersight if you’ve been logged out. The instructions for logging into OKTA are located at this link: https://fso.cisco-one.com/okta/

🔵 From the pull-down menu, select System.

🔵 Next use the steps below to claim a new target for your EKS Kubernetes cluster.

  1. Click on the ‘Admin’ tab on the left menu
  2. Click on the ‘Targets’ tab
  3. Click on the ‘Claim a New Target’ button

🔵 Now use the steps below to select the Kubernetes target type.

  1. Click on the ‘Cloud Native’ category
  2. Click on the ‘Kubernetes’ tile
  3. Click on the ‘Start’ button

🔵 Use the steps below to finish creating your target.

  1. Enter the Device ID you saved for the IWO collector running in EKS in the Device ID field
  2. Enter the Security Token you saved for the IWO collector running in EKS in the Claim Code field
  3. Click on the ‘Claim’ button

Claim Target for IKS
(Reference Only)

While the steps to claim the Kubernetes target for the IKS cluster are defined below, these steps have already been done for you in this workshop. The steps are here for your reference so you have all the steps documented that were part of the complete IWO setup. Do not do these steps.

🔵 Use the commands below to verify the deployment of the IWO collectors in the IKS cluster.

kubectl get pods -o wide -n iwo-collector –kubeconfig $iks_kubeconfig_filepath

🔵 Highlight and copy the name of the first pod in the list of pods returned for the IKS cluster and save that pod name for the next step

🔵 Use the command below to access the IWO collector running in IKS. Use the collector pod name running in IKS that you obtained previously to substitute {your-iks-pod-name-saved-previously} in the command below.

kubectl -n iwo-collector port-forward {your-iks-pod-name-saved-previously} 9110 –kubeconfig $iks_kubeconfig_filepath

The output should look like the image below.

🔵 Open a new terminal window in Cloud9 and use the command below to return the Device ID for the IWO collector running in IKS.

curl -s http://localhost:9110/DeviceIdentifiers

Now highlight the Device ID (without including the double quotes) and save that ID in a text file.

Next use the command below to return the Security Token for the IWO collector running in IKS.

curl -s http://localhost:9110/SecurityTokens

Now highlight the Security Token value (without including the double quotes) and save that value in a text file.

🔵 Use the credentials for Cisco Intersight via OKTA provided by your instructor to login to Intersight if you’ve been logged out. The instructions for logging into OKTA are located at this link: https://fso.cisco-one.com/okta/

🔵 Next use the steps below to claim a new target for your IKS Kubernetes cluster.

  1. Click on the ‘Admin’ tab on the left menu
  2. Click on the ‘Targets’ tab
  3. Click on the ‘Claim a New Target’ button

🔵 Now use the steps below to select the Kubernetes target type.

  1. Click on the ‘Cloud Native’ category
  2. Click on the ‘Kubernetes’ tile
  3. Click on the ‘Start’ button

🔵 Use the steps below to finish creating your target.

  1. Enter the Device ID you saved for the IWO collector running in IKS in the Device ID field
  2. Enter the Security Token you saved for the IWO collector running in IKS in the Claim Code field
  3. Click on the ‘Claim’ button

Claim Target for AWS

(Reference Only)

While the steps to claim the AWS target for the EKS cluster are defined below, these steps have already been done for you in this workshop. The steps are here for your reference so you have all the steps documented that were part of the complete IWO setup.

🔵 Use the credentials for Cisco Intersight via OKTA provided by your instructor to login to Intersight if you’ve been logged out. The instructions for logging into OKTA are located at this link: https://fso.cisco-one.com/okta/

🔵 Next use the steps below to claim a new AWS target for your EKS Kubernetes cluster.

  1. Click on the ‘Admin’ tab on the left menu
  2. Click on the ‘Targets’ tab
  3. Click on the ‘Claim a New Target’ button

🔵 Now use the steps below to select the Amazon Web Services target type.

  1. Click on the ‘Cloud’ category
  2. Click on the ‘Amazon Web Services’ tile
  3. Click on the ‘Start’ button

🔵 Use the steps below to finish creating your target.

  1. Enter a descriptive name that represents you AWS target based on the context your using it for (IWO)
  2. Enter your AWS Access Key
  3. Enter your AWS Secret Access Key
  4. Click on the ‘Claim’ button

Claim Target for AppDynamics

Reference Only

While the steps to claim the AppDynamics target are defined below, these steps have already been done for you in this workshop. The steps are here for your reference so you have all the steps documented that were part of the complete IWO setup.

🔵 Use the credentials for Cisco Intersight via OKTA provided by your instructor to login to Intersight if you’ve been logged out. The instructions for logging into OKTA are located at this link: https://fso.cisco-one.com/okta/

🔵 Next use the steps below to claim a new AppDynamics target.

  1. Click on the ‘Admin’ tab on the left menu
  2. Click on the ‘Targets’ tab
  3. Click on the ‘Claim a New Target’ button

🔵 Now use the steps below to select the AppDynamics target type.

  1. Click on the ‘Application Performance Monitoring (APM)’ category
  2. Click on the ‘AppDynamics’ tile
  3. Click on the ‘Start’ buttonA

🔵 Use the steps below to finish creating your target.

  1. Switch the ‘Connect through an Intersight Assist’ toggle button to the left so that it is off
  2. Enter the host name of your AppDynamics controller (without protocol or port)
  3. Enter the username from your controller that has Admin level privilages
  4. Switch the ‘Secure’ toggle button to the right so that it is on
  5. Enter the listener port of your AppDynamics controller
  6. Enter the password for the user name entered in step three
  7. Click on the ‘Claim’ button

FINALIZE SETUP

In this section we’ll verify the setup steps you performed and confirm that data is being collected by the different agents you deployed.

Verify AppDynamics Agents

🔵 Use the steps below in your Cloud9 IDE to find your login details for the AppDynamics controller.

  1. On the left-hand navigation tree expand the ‘environment/workshop’ directory
  2. Double-click on the ‘workshop-user-details.txt’ file to open it
  3. Use the ‘Controller URL’ in the file to navigate to the AppDynamics controller in your browser

Enter the Account, Username, and Password found in the file to login to the AppDynamics controller.

🔵 Use the steps below to navigate to your TeaStore application.

  1. Click on the ‘Applications’ tab on the top menu
  2. Find your TeaStore application {cluster-prefix}-teastore-fso-lab-{lab-number} and click on its name to open 

🔵 Use the steps below to verify that Tiers & Nodes are registering with the controller.

  1. Click on the ‘Tiers & Nodes’ tab on the left menu
  2. Click on the Grid View button
  3. Check to validate that 6 nodes have registered
  4. Click on the Refresh button until you see all 6 nodes

If you don’t see all 6 nodes within ~5 minutes you can move on to the next verification steps and come back after that to check that all 6 nodes have registered.

🔵 Use the steps below to verify that the Cluster Agents are registering with the controller.

  1. Click on the ‘Servers’ tab on the top menu
  2. Click on the ‘Clusters’ tab on the left menu
  3. Type in {cluster-prefix}-teastore-fso-lab-{your-lab-number} in the search box
  4. Verify that you see 2 Cluster Agents, one for EKS and one for IKS

Configure Secure

Application Policy

🔵 Use the steps in the previous exercise to navigate to the TeaStore application with your lab number in the name in the AppDynamics controller.

  1. The application should open on the ‘Application Dashboard’ tab
  2. Now click on the ‘Security Health’ link to open the Secure Application UIHealth

🔵 Create a new Policy in Secure Application by using the steps below.

  1. Click on the configure icon on the top menu
  2. Now click on   ‘Policies’  on menu that appears.
  1. Click on the ‘Create New Policy’ button on the right

🔵 Enter the details of the new Policy by using the steps below.

  1. Select Network or socket access (NETWORK)
  2. Type in {cluster-prefix}-teastore-fso-lab-{your-lab-number} in the ‘Application’ box and select your TeaStore Application from the drop-down
  3. The ‘Tier’ field should be set to All
  4. The ‘Default Action’ field should be set to Detect
  5. In the ‘Rules’ area:
    • Click the + Rule link to add a new rule
    • Select stack trace and contains in the first two fields
    • Enter the string org.apache.logging.log4j.core.lookup.JndiLookup.lookup in the third field
    • Select Block in the fourth field
  6. Click on the Save button

🔵 Click on the Save button to confirm you want to create the Block policy.

Verify IWO Collectors

🔵 Use the credentials for Cisco Intersight via OKTA provided by your instructor to login to Intersight if you’ve been logged out. The instructions for logging into OKTA are located at this link: https://fso.cisco-one.com/okta/

  1. Select “Workload Optimizer” from the pull-down menu.

🔵 Use the credentials for Cisco Intersight via OKTA provided by your instructor to login to Intersight if you’ve been logged out. The instructions for logging into OKTA are located at this link: https://fso.cisco-one.com/okta/

  1. Click on the ‘Business Application’ icon at the top of the Supply Chain.

🔵 Find your specific Business Application using the steps below.

  1. Find the Business Application that has your lab number in the name and verify it is in the list
  2. If needed click the ‘Show All’ link to find your Business Application and verify it is in the list.
Next Section