Security

FSO SECURITY

In this “Better Together” story let’s explore how Cisco Secure Application and AppDynamics combined capabilities enable SecOps teams to gain a deeper visibility into vulnerabilities, runtime detection of exploits, and real-time defense against attacks.

Learning Objectives

  • Understand how Cisco Secure Application and AppDynamics bring more value together
  • Get hands on experience on how Cisco Secure Application extends the current performance monitoring capabilities of AppDynamics
  • Explore scenarios where vulnerabilities, runtime detection of exploits, and real-time defense against a broad range of attacks can be quickly analyzed and triaged

Who Can Benefit?

  • DevOps: By enabling a tight feedback loop for the software delivery life cycle
  • AppOps: By improving triage capabilities for the internet and deployment environments

APPLICATION SECURITY

The penalty for a security breach is high. A lack of focus on cybersecurity can be incredibly damaging to an organization. There is a direct financial impact such as theft of confidential information, disruption to trading or even having to repair affected systems all resulting in financial loss. Additionally, cybersecurity breaches also cause reputational damage which sometimes has far bigger impact than just the financial loss and can be fatal to an organization.

The moment an application is deployed to production, new security holes and zero-day exploits make the Application vulnerable despite all the efforts of the pre-production testing. In a siloed world where Security Team is focussed on Vulnerabilities and Threats, and App Team is focussed on Application Topologies, Jill (Sr. Director, Cyberdefense, Bridge to Possibili-Tea) wondered if it was possible to have a Security Solution that could bring multiple teams together and reduce MTTD and the MTTR.

Software engineers working on project and programming in company

A New Approach

With applications running anywhere, changing constantly, and being unique, Jill needed to take a new approach to security. She wanted a solution that offers ‘Inside-Out Security’. It should:

  • Be closer to the application runtime
  • Continuously evaluate the Application for security
  • Share context with different teams based on metrices from Application Performance Monitoring (APM) and Business Intelligence (BiQ) and
  • Be adaptive for the Application

WHAT KEEPS JILL AWAKE

Jill (Sr. Director, Cyberdefense, Bridge to Possibili-Tea), is well aware of the penalty for a security breach. A lack of focus on cybersecurity can be incredibly damaging to her organization. There is a direct economic impact such as theft of confidential information, disruption to trading or even having to repair affected systems all resulting in financial loss. Additionally, cybersecurity breaches also cause reputational damage which sometimes has far bigger impact than just the financial loss.

Software engineers working on project and programming in company

Jill can’t get back the time it takes her team to detect and contain an active breach. Rather than just saying “I found something at this URL, can you fix it?”, Jill wanted to go to the DevOps team with exact data points and tell them – “I found vulnerability CVE–XXXX-XXXXX in the ‘Checkout’ Service Tier of the Application ‘TeaStore’; here is the details of the vulnerability with adversary tactics and techniques; this is how bad (severe) it is based on the Common Vulnerability Scoring System (CVSS) score and here are the remediation steps.” This kind of information can be a huge time saver for the DevOps team.

Jill and her team want to work and solve the following pain points:

Reduce the time to detection from days/months to minutes
Protect the application against attacks in real time
Identify vulnerabilities and threats within the applications in production and suggest suitable remediation steps
Reduce the attack surface by collaborating efficiently with other teams to fix the vulnerabilities and mitigate the threats
To share the security context that’d help in prioritizing remediation tasks based on business impact

WHY CISCO

Why Cisco Secure Application?

After evaluating many Application Security products, Jill and her team decided to implement Cisco Secure Application because of these reasons:

🔵 Runtime detection

Typically, vulnerability scanning is done pre-production or scheduled monthly/quarterly etc. But Cisco Secure Application provides real-time production assessment of vulnerabilities by scanning code execution to prevent exploits regardless of where the application is hosted or where the traffic originated and then maps them back to the running instances. It detects application code dependency and security vulnerabilities in production. The business can operate at a faster pace with a lower risk profile due to constant run-time protection, real-time remediation, and security automation.

🔵 Significant reduction in MTTD and MTTR

Protection at runtime gives visibility into application’s true behaviour and you can catch zero-day exploits in minutes and not in weeks/months. It gives the details of vulnerabilities, their severity levels, the remediation steps that includes the details of the running part of the code that is vulnerable and what to change in to mitigate the risk. This reduces the Mean Time to Detection from days/weeks to minutes and helps in drastically reducing the time it takes to remediate.

🔵 Automatic & continuous

It simplifies the life cycle of vulnerability fixes and security incidents. New discoveries and resolutions are tracked and reported without having to wait.

🔵 Strategic prioritization based on Business Impact

Vulnerability data is correlated to the application and business context so that application and security teams can prioritize, execute, and track remediation by business impact

🔵 No more siloed teams

With Secure Application, application and security teams can leverage correlated performance and security insights to minimize their organization’s attack surface, safeguard customer data, keep business impact and objectives top of mind, and thrive in today’s digital environment.

🔵 Secures modern applications without sacrificing velocity

Secure Application reduces the risk of security exposure without compromising the speed of delivery.

🔵 Anomaly detection and prevention

Provides visibility into Application’s true behaviour and detects deviations from the normal. Automatically enforces security policies to block the threats.

🔵 No additional Agent

There is no need of deploying any additional agent (It’s embedded into existing APM agent). It has negligible overhead and doesn’t impact end-users.

🔵 Works with existing Security Solutions

It compliments existing security products such as WAFs, NGWAFs etc. and works well with products like Cisco Secure Cloud Analytics, Cisco Secure Workload and Cisco Secure Access (Duo) to protect the Application from ‘Inside’ and ‘Outside’.

Secure Application delivers substantial value to the Organizations by pairing AppDynamics with the strength Cisco has in the security market, making security an integral part of the enterprise application monitoring strategy. It extends the current performance monitoring capabilities of AppDynamics to provide continuous monitoring for vulnerabilities, runtime detection of exploits, and real-time defense against a broad range of attacks.

Thanks to Secure Application, Jill’s organization Bridge to Possibili-Tea can protect applications at runtime, detect and block attacks in real-time, and simplify the lifecycle of security incidents by providing application and business context and creating a shared context across Application and Security teams to drastically reduce the MTTD and MTTR. Application Team can now scope a security incident and understand the potential impact to users and the business and strategically prioritize remediation by business impact.

LAB ENVIRONMENT

For Cisco Secure Application, there is no need to deploy any additional agent. As shown in the diagram below, the feature is embedded into existing APM agent. It has negligible overhead and doesn’t impact end-users.

Though not relevant to this lab exercise, Cisco Secure Application works well alongside other security solutions like WAF, NGWAF, NGFW, etc, that protect at the perimeter.

HANDS-ON OBSERVABILITY

Bill is the Lead SecOps Engineer who works for Jill at Bridge to Possibili-Tea. You will be working with Bill during this exercise. Your job is to make sure Bill achieves his goals for which Bridge to Possibili-Tea bought Cisco Secure Application.

Testing software

Bill wants to secure his application against the log4j CVE-2021-44228 vulnerability that can result in remote code execution giving complete control over the application and underlying system.

To achieve the above, we will take Bill through the following steps:

  1. Access Business Risk Profile
  2. Detect if the vulnerability (log4j CVE-2021-44228) is actually present in the application
  3. If it is, block the attacks without even updating the code or restarting the app
  4. Making sure the attacks were being blocked
  5. Further investigating the attacks and the related vulnerability across all tiers of the application
  6. Suggesting ways for a permanent remediation for this vulnerability

UNDERSTANDING

SECURITY STATE

Bill wants to have actionable information. In this lab, you will make sure Bill has a quick way to find how his Applications are doing from security point of view. Bill wants to be comfortable while navigating the Dashboard so he knows what kind of information is readily available to him.

🔵 Use the credentials used previously to login to the AppDynamics controller if you’ve been logged out.

AppDynamics Application Dashboard

🔵 Use the steps below to navigate to your TeaStore application.

  1. Click on the ‘Applications’ tab on the top menu
  2. Find your TeaStore application with your lab number in the name and click on its name to open it

🔵 The Application Dashboard displays showing the components that comprise the TeaStore application.

  1. The ‘Security Health’ widget shows us if we have any Critical or Warning events
  2. Click on the ‘Security Health’ link to launch the ‘Secure Application’ dashboard

CSA Home Dashboard

🔵 The Home page is the landing page of the Dashboard and it provides a high-level overview of Attacks and Vulnerabilities of the monitored application.

  1. The Vulnerabilities pane shows the number of both Open and Fixed vulnerabilities. The Pie chart in the left represents the total number of open vulnerabilities.

    • The color of the Column chart displays the number of vulnerabilities for different severity levels:
      • Critical = Red   High = Pink   Medium = Orange   Low = Yellow
    • Days Since First Detected shows the number of days the vulnerability has been open for and it’s severity

  2. The Attacks pane displays the number of open attacks, Top Applications and Top Attack Types.

    • The Column chart shows the total number of Open attacks and their states:
      • Exploited = Dark Blue   Blocked =  Blue   Attempted = Light Blue
    • Top Attack Types displays the top 10 attack events in different states (exploited, blocked or attempted) against the total number of open attacks
    • Trend shows the trend of fixing the open vulnerabilities

🔵 Scroll down the screen to see the Observations and Applications pane.

  1. The Observations pane shows top 10 Applications and Event Types that are observed with malicious activities

  2. The Applications pane displays the nodes of the application in question and the trend

    • Total Active Nodes is the number of nodes that are actively communicating with the Controller
    • Supported Nodes is the number of nodes registered/unregistered with Cisco Secure Application
    • Enabled Nodes is the number of nodes that have security enabled/disabled for the applications
    • Secured Nodes is the number of secured/unsecured nodes
    • Trend displays the number of enabled, supported, active, and secured nodes against the day of the month

Summary

🔵 Just by looking at the Home page of his Secure Application Dashboard, Bill has not only learned about his application and it’s security state but he also has actionable information.

  • Number of Open Vulnerabilities, their severity levels and the lifespan of those open vulnerabilities
  • Open attacks that needs to be fixed and their states
  • Top Applications with open attacks and event types
  • Nodes of the application(s) that needs to be enabled and secured

ATTACK DETECTION PREVENTION

In the previous lab (‘Understanding Security State of an Application’) you obtained some actionable information and learnt how the Dashboard made it easy to understand the state of an application. We will explore this further in the following lab (‘Business Risk Profiling and Prioritization’). But, now that we know we have some open vulnerabilities that carry critical and high severity levels, our immediate priority should be to make sure adversaries are not able to exploit these vulnerabilities or make any further damage.

 

Learning Objectives

In this lab you will:

Detect if the vulnerability (log4j CVE-2021-44228) is actually present in the application
If it is, block the attacks without even updating the code or restarting the app
Confirm that the attacks are being blocked

Vulnerabilities Dashboard

🔵 Use the credentials used previously to login to the AppDynamics controller if you’ve been logged out.

🔵Use the steps you used previously to navigate to the Secure Application Dashboard if you’ve been logged out.

🔵 Let’s go to the vulnerabilities tab as shown below to find out if the vulnerability (log4j CVE-2021-44228) is present in your TeaStore application.

  1. Click on the Vulnerabilities tab that shows number of Open vulnerabilities, their severity levels and lifespans
  2. Observe all the vulnerabilities, their severity levels (critical, high, medium, low)
  3. In this column you see the specific types of vulnerabilities that have been detected
  4. Notice the application that is vulnerable (in our case it is only our TeaStore app as we have restricted the scope)
  5. The Status column tells us if a particular vulnerability has been Discovered, Confirmed, or Fixed yet

🔵 Let’s look at the specific vulnerability we are interested in and focus on Detecting it and then Preventing the adversaries from exploiting it.

  1. Select ID from the filter box drop-down
    • Type in the string 44228
    • Select CVE-2021-44228 from the drop-down
  2. This confirms a Critical vulnerability (CVE-2021-44228) is present in our application
  3. The Discovered status means Bill needs to do something urgently as it carries a Critical vulnerability
  4. Tier (Nodes) column tells us the affected services or Tiers in the application that are vulnerable
  5. Click on the CVE-2021-44228 link under Vulnerability column, taking you to the vulnerability detail page

🔵 Now we can see more detail about the vulnerability and possible remediation steps for it.

  1. The ‘Overview’ box gives an overview of this vulnerability. Scroll down the overview box to learn more.
    • It describes how attackers exploit this vulnerability and further detail on suggested remediation steps
  2. Look at the First/Last Seen details
  3. The Remediation area lists the suggested framework versions that would not have the vulnerability
  4. You can click on the external link next to ‘CVE-2021-44228’ in the ‘ID’ area
    • This will take you to the synk Vulnerability DB website where you can learn more about this vulnerability (as seen in the next image)

🔵 Example of the synk website for ‘CVE-2021-44228’

Attacks Dashboard

Now that we know we have a vulnerability present in our environment and how bad it is, let’s make sure our application is protected against this vulnerability. During the Lab Environment Setup you created a Policy within CSA to block the ‘CVE-2021-44228’ vulnerability. Now, let’s make sure this policy is doing it’s job and blocking the attacks.

🔵 Use the steps below to look at the details of the blocked ‘CVE-2021-44228’ vulnerability.

  1. Click on the Attacks tab at the top of the Secure Application screen
  2. Select Outcome from the filter box drop-down
    • Then select Blocked from the box drop-down
    • The screen will be populated with all the Attacks that have been Blocked
  3. Click on the first attack in the list

🔵 Now we are looking at the Attack Details screen.

  1. Observe the Timestamp and the Affected Node
  2. Look at the Entry Point and Client IP where the attack originated from
    • This is vital information
  3. Now click on Show More link next to the Stack Trace

🔵 Inspect the Stack Trace from the attack that was blocked.

As highlighted below, it confirms a line with ‘org.apache.logging.log4j.core.net.JndiManager.lookup’. This information can be used to guide developers to the lines of code that was used to achieve the result of the event.

Summary

In this Lab, you confirmed the vulnerability log4j CVE-2021-44228 was present in your environment and how severe it was, you then created a policy with a rule to block the attacks that exploit this vulnerability. You went ahead and confirmed that the rule you created was actually effective in blocking those attacks.

RISK PROFILING PRIORITIZATION

In the lab ‘Attack Detection and Prevention’, you detected the vulnerability in your environment and then observed that the policy you created previously blocked the Attack that leverages a particular vulnerability. You made sure that the policy worked as expected. You did well as this step bought some valuable time for your DevOps and SecOps Teams to plan and implement a permanent remediation method.

Learning Objectives

In this lab you will:

  • Understand the Risk impact of the vulnerability on your business
  • Based on the Risks and the Impact, understand the prioritization plan of the AppSec Team so the vulnerability can be remediated permanently

Libraries Dashboard

  1. 🔵 Use the credentials used previously to login to the AppDynamics controller if you’ve been logged out.

🔵 Use the steps you used previously to navigate to the Secure Application Dashboard if you’ve been logged out.

🔵Now, let’s’ explore the Libraries that have been detected using the steps below.

  1. Click on the ‘Libraries’ tab on the top menu
  2. Click on the ‘Risk’ column header until you see the most severe vulnerabilities at the top of the list. 
  3. Here we see the Type of the Application (Java in this case) and which Tier or Service is using a particular Library.
  4. Secure Application also suggests the version of the library (Remediation Candidate) to remediate those vulnerabilities in the respective LibrariesRisk

In the above example let’s say your ‘teastore-webui’ service is the bread and butter of your business and you can’t afford any downtime with this Tier. It also has a Risk score of 33.1 with two critical vulnerabilities. This must be considered while deciding and prioritizing your remediation efforts.

Summary

In this lab, you learned how much risk a particular Library carries. It would immensely help your DevOps Team in deciding where to start from. You quickly learned which library need to be replaced first and how much benefit might come from it.

Well Done again! It was a quick lab but you now have the information for your DevOps Team so they can prioritize their remediation efforts.

VULNERABILITY

ASSESS REMEDIATE

Benefitting from the last lab ‘Business Risk Profiling and Prioritization’, you are now aware of the vulnerability that poses the highest risk to your business. This would allow you to suggest the library that should be replaced first.

Learning Objectives

You now pass the information gathered during the last lab on to your DevOps Team. This will immensely help them in deciding where to start from.

Continuing on in this lab you will:

  1. Further assess the vulnerabilities that a particular Library carries
  2. Suggest ways to permanently remediate this vulnerability
  3. Make sure the DevOps Team have the information that helps them in determining how much effort it will take to permanently remediate a vulnerability

Libraries Dashboard

🔵 Use the credentials used previously to login to the AppDynamics controller if you’ve been logged out.

🔵  Use the steps you used previously to navigate to the Secure Application Dashboard if you’ve been logged out.

🔵 Now, let’s’ explore the Libraries dashboard again so you can provide your recommendation and level of effort to remediate these vulnerabilities to the DevOps Team.

  1. Click on the ‘Libraries’ tab on the top menu
  2. Select ‘Libraries’ from the filter box drop-down
    • Type in the string log4j-core
    • Select org.apache.logging.log4j:log4j-core:2.14.0 from the drop-down
  3. The ‘teastore-webui’ service is the front end UI where your customers order your products and your business can’t afford any downtime with this
  4. It carries a Risk score of 10.0 which is one of the highest scores
    • It also has a total of four vulnerabilities including two with ‘Critical’ severity level
  5. Click on this Library (shown below) to explore it further

🔵

Let’s look at the following information viewing the details screen for the library.

  1. It shows all the known vulnerabilities that are present in the application
  2. Each of the vulnerabilities is shown with it’s ‘Severity Level’ and ‘Impact Score’
    • (hover over the severity level circle to see the impact score)
  3. The ‘Remediation Candidate’ column shows the lowest version needed to fix a particular vulnerability
    • This helps to prevent the rest of the application code from breaking unnecessarily by upgrading the library “too high” just while dealing with a security issue

With this information in hand, the DevOps team now knows exactly where to start from, which vulnerability to fix first and what is the lowest version of remediation candidate for each of those vulnerabilities. This helps them vastly in planning their remediation efforts and offering a permanent fix without breaking rest of the application.

🔵 Optional Clicking on any of those vulnerabilities will display you a similar screen as shown below.

You see the details about that vulnerability with it’s severity level and Risk Score. You also learn when it was detected first and what it’s current status is. We did talk about all this while doing the lab ‘Attack Detection and Prevention’.

Summary

Knowledge gained during this lab could be of great help to your DevOps Team when they plan to permanently remediate the vulnerabilities present in the applications. This will ensure they put in minimum efforts and don’t take unnecessary risks with the rest of the application code.

 

We’re sure you’ve gained some valuable insight along with practical skills today!

WORKSHOP CLEANUP

Now it’s time to do a small amount of cleanup that will only take a few steps to complete.

Delete CloudFormation Stack

🔵 Use the link below in a new browser tab to navigate to CloudFormation in the AWS console.

https://us-west-1.console.aws.amazon.com/cloudformation/home?region=us-west-1

🔵 Use the steps below to find your CloudFormation stack and delete it.

  1. Type in {cluster-prefix}-FSO-Lab-{your-lab-number} in the search box and hit enter
  2. Select your CloudFormation Template
  3. Click on the Delete button

🔵 Now click on the Delete stack button to confirm deletion of your CloudFormation stack.

Run Workshop Teardown

🔵 In your Cloud9 terminal use the commands below to teardown the remaining workshop artifacts.

cd /home/ec2-user/environment/workshop

./teardownWorkshop.sh

The output from the teardown script when it starts should look like this and should take ~8 minutes to complete.

The output from the teardown script when it finishes should look like this.

Awesome efforts! You have now completed the last Lab of this course. Thank You for your time!