Zero Trust AWS GWLB QwikLab Guide

Overview

The goal of this workshop is to take you through a deployment scenario where the Palo Alto Networks VM-Series firewall is deployed on an AWS environment. We will discuss design scenarios leveraging the AWS Gateway Load balancer. This workshop will demonstrate a use case that involves deploying Zero Trust Policies on VM-Series Virtual Next-Generation Firewall (NGFW) to protect your applications against various attacks.

Hands-On Lab – Zero Trust Exercise Summary

As part of this workshop, you will be deploying security services on Palo Alto VM-Series Virtual Firewalls to protect your virtual application environment against the Log4j attack.  You will follow best practice recommendations to deploy a tiered protection against Log4j and similar attacks with the VM-Series NGFW.

A Zero Trust implementation should provide security administrators with the visibility and the ability to secure traffic between the various applications. 

Hands-On Lab – Palo Alto Networks Product Coverage

• VM-Series Virtual Next-Generation Firewalls

  • Protect Private Cloud investments
  • Safeguard cloud and virtualized investments with the virtual firewall that pays for itself. VM-Series NGFWs are built for high performance and significant ROI.
  • Detect hard-to-find threats, Stop outbound traffic exfiltration, Protect against lateral movement

• Panorama™

  • Consolidate policy management. Panorama™ network security management simplifies policies across infrastructures and clouds. Seamless integration. Increased oversight.
  • Panorama can be deployed as a virtual appliance on VMware ESXi™ and vCloud® Air, Linux KVM, and Microsoft Hyper-V®.

Application Environment Overview

Figure: Hands on Lab Environment – Achieving Zero Trust Security

For this workshop, we have pre-deployed some components for you.  You will see a VM-Series, Panorama and an Application Virtual Machine and an Attacker VM pre-deployed.  

What You'll Do

• Use Zero Trust Policy on VM-Series to protect your application against Log4j attack 
• Use Panorama to apply Zero Trust Policy on VM-Series NGFWs
• Use Panorama to Monitor Logs to see Security activity

Launch the lab environment

In this section, we will launch the lab environment. These are the steps that we will accomplish at this time.

• Start the lab on your designated Qwiklab account.
• Login to the AWS Console using the provided credentials and set up  IAM roles
• Subscribe to the Palo Alto Networks VM-Series PAYG and Panorama appliances on the AWS Marketplace.
• Deploy lab environment using Terraform

Start qwiklabs lab environment and login to AWS

1. Log in to the Qwiklabs portal and click on the "Zero Trust AWS GWLB lab" on the home page. If you do not have access to the lab, please reach out to the lab instructor to get the access.
2. On the Qwiklab environment, Click on Start Lab Button to start the lab.

At this point, Qwiklabs will build an AWS account for you. You will find the login credentials for accessing the AWS account under the Open Console button as shown below.

3. To login to the AWS environment, right click on “Open Console’ and “Open link in Incognito window” for Chrome-based browsers. For other browsers, use the appropriate option to open the AWS Console in a new private tab.

4. On the AWS Console, copy over the IAM username and password from the Qwiklabs page.

5. Now, click on “Sign In”.

Once you are successfully logged in, you will land on the AWS Management Console.

Figure: The AWS Management Console

Set up AWS Account permissions

As mentioned earlier, the Qwiklab user account, by default, does not have the permissions to AWS Marketplace and CloudShell services, which are required for the purpose of this lab. We will now edit the permissions for the Qwiklab user account to provide access to those services.

6. On the AWS console, in the search bar at the top, type ‘iam’.
7. Click on the link to IAM. A new IAM dashboard window will open.

8. From the left menu, select Users and from the list of users displayed, click on awsstudent.

9. Under Permissions policies, Click on default policy to edit the default permissions policy.

10. Click the ‘JSON’ tab.

11. Scroll down to the Deny policy and remove two lines (line number 27 and line number 36) listed below

"aws-marketplace:\*ubscribe",
...
"cloudshell:\*",

Make sure to delete the whole line.

12. Click on ‘Review policy’ at the bottom of the screen.

13. On the next screen, click on ‘Save changes’.

14. Account setup is now complete.

Launch lab environment

In this section, we will deploy the AWS Cloud resources required for the purpose of this lab using Terraform. 

15. Click on ‘AWS’ on the top left hand corner to navigate to the primary console.

Subscribe to VM series

16. Click on the link below and the page that opens up, Click on “Continue to Subscribe” and then Click on “Accept Terms”.

Make sure to open the below link on the same browser/window to ensure that you are already logged in to the same AWS account.

https://aws.amazon.com/marketplace/pp?sku=hd44w1chf26uv4p52cdynb2o

17. Wait a few minutes until the effective date changes from  ‘Pending’ to date.

Subscribe to Panorama

18. Click on the link below and the page that opens up, Click on “Continue to Subscribe” and then Click on “Accept Terms”.

Make sure to open the below link on the same browser/window to ensure that you are already logged in to the same AWS account.

https://aws.amazon.com/marketplace/pp?sku=eclz7j04vu9lf8ont8ta3n17o

19. Wait till Effective date changes from ‘Pending’ to today’s date. This will take a few minutes.

Deploy AWS Lab resources

20. From the AWS management Console, launch CloudShell using the icon on the top right side of the console.

If you do not see the icon as shown in the image above, you can search for CloudShell on the Search bar on top to open the service.

21. Close out the welcome pop up.

It takes around a minute for cloudshell to launch and to get the prompt as shown in the example below.

22. After the cloudshell is launched, copy and paste the below commands on the shell prompt and hit the Enter key to execute them.

rm -rf * && git clone https://github.com/PaloAltoNetworks/lab-aws-zero-trust.git

lab-aws-zero-trust/setup.sh

It will take a few minutes  (~5m ) to deploy all the lab components. Status will be updated on the cloudshell console as deployment progresses. At the end of deployment, you should see the message “Completed successfully!”

Figure: Completion message of the Lab Setup script

If you run into any errors during the setup script execution, please check the Troubleshooting section on how to resolve the errors.

23. Review the deployed lab environment with the topology diagram shown below.

Figure: The Lab topology that is deployed using Terraform

Log in to Panorama

24. Run the below command to get the Panorama URL.

cd ~/lab-aws-zero-trust/terraform/vmseries && terraform output PANORAMA_IP_ADDRESS && cd

25. Copy and paste the Panorama URL on a new browser tab and hit the Enter key. On the Login page, enter the credentials as shared below.
    • Username: admin
    • Password: Paloalto@1

26. Once logged in, navigate to the Panorama tab, Click on Managed Devices > Summary on the vertical menu. Ensure that the VM-Series NGFW is in Connected state.

If you face some issues accessing the Panorama Web UI or logging into Panorama or post login if you are unable to find the VM-Series NGFW connected to the Panorama, please wait for a couple more minutes for Panorama to be setup completely.

Make sure that the VM-Series NGFW is in Connected state and In Sync with the Panorama. If it is not, you will not be able to login to the spoke servers.

Set up the Vulnerable application server

27. On CloudShell, open a new CloudShell tab by clicking on the Actions menu on top and selecting New tab.
28. Run the below command to connect to the vulnerable server, vul-app-server.

aws ec2-instance-connect ssh --instance-id $(aws ec2 describe-instances --filters "Name=tag:Name,Values=qwikLABS-vul-app-server" "Name=instance-state-name,Values=running" --query 'Reservations[].Instances[].[InstanceId]' --output text)

There are also other ways to connect to the EC2 instances, you can look them up in the Appendix section

29. Verify that the vulnerable application is already running by running the below command.

sudo docker container list -a

30. In this lab environment we do not have a DNS set up. So we would add an entry directly on the pod hosts file.

sudo docker exec -it vul-app-1 /bin/sh -c 'echo "10.2.1.100 att-svr" >> /etc/hosts'

31. To verify the update of the hosts file, try to ping the att-app-server using the hostname (att-svr) provided. 

sudo docker exec -it vul-app-1 /bin/sh -c 'ping att-svr'

Launch Log4J Attack

32. On CloudShell, open a new CloudShell tab by clicking on the Actions menu on top and selecting New tab.
33. Run the below command to connect to the attack server, att-app-server.

aws ec2-instance-connect ssh --instance-id $(aws ec2 describe-instances --filters "Name=tag:Name,Values=qwikLABS-att-app-server" "Name=instance-state-name,Values=running" --query 'Reservations[].Instances[].[InstanceId]' --output text)

34. Verify that the attack application is already running by running the below command.

sudo docker container list -a

35. On the att-app-server command prompt, execute this command to launch the Log4J attack.

/tmp/launch_attack.sh

36. You should receive a ‘Hello World’ message back.

37. The vulnerable application has now been infected by malware. Go back to the SSH session with vul-app-server and execute the below commands:

sudo docker exec -it vul-app-1 /bin/sh

ls -lrt /tmp

For example:

38. On your Panorama tab on the browser, navigate to Monitor > Traffic

 

39. Notice the 3 sessions associated with a successful Log4J attack.

Source	Destination	Destination Port	Application
att-svr(10.2.1.100)	vul-app(10.1.1.100)	8080	web
vul-app(10.1.1.100)	att-svr(10.2.1.100)	1389	LDAP
vul-app(10.1.1.100)	att-svr(10.2.1.100)	8888	Malware transfer

40. Remove the malware-sample file for subsequent tests.

rm -rf /tmp/malware-sample

Best Practices for Protection against Log4J

In the following sections, we will see how you can build a policy model that will provide multi-tiered protection for your application against Log4j attack. Multiple complementary security controls across our portfolio, combined with best practices, can help protect organizations against the vulnerability. If your organization is already aligned with our security best practices, you gain automatic protection against the multiple steps of this attack with no manual intervention. 

In the following section, we will deploy best practices zero trust policies to  protect against attacks. We will deploy 3 policies one each for outbound, inbound and application based protection.

Deploy Zero Trust Policy for INBOUND Traffic

41. On the Panorama tab on the browser, navigate to Policies tab. Ensure that you are in Shared Device Group

The policies will look like this.

42. Enable ZT-log4j_BLOCK_INBOUND

43. On the policy, Click on ZT-log4j_BLOCK_INBOUND, navigate to the ‘Profile’ column and click on the icon shown there. Ensure that ‘Strict’ vulnerability protection profile has been enabled for this policy. 

44. Click ‘OK’ and commit the changes and push to device.

If you see the message "Not all Commit jobs got triggered", see this section on how to resolve this issue.

45. Click in ‘Tasks’ button on the bottom horizontal bar and ensure that ‘commit all’ job finishes. Click on Close to exit out of the window.

46. Relaunch the attack by navigating to the SSH session on att-app-server and issue the command.

/tmp/launch_attack.sh

47. This time we will see it does not go through

48. Go back to the Panorama UI and navigate to Panorama > Monitor > Threat
49. You will see that the Firewall detected the attack and blocked it.

It may take a minute for the logs to show up on the Panorama.

Deploy Zero Trust Policy for OUTBOUND Traffic

The Log4j attack requires access to malicious Java code hosted externally. Our Advanced URL Filtering and DNS Security service are constantly monitoring and blocking new, unknown and known malicious domains (websites) to block those unsafe external connections.

In this activity, we will use a policy that has both the profiles set to protect outbound traffic.

50. On the Panorama tab on the browser, navigate to Policies tab. Ensure that you are in Shared Device Group
51. Disable policy 1 and Enable policy 2. 

52. On the policy, Click on ZT-log4j_BLOCK_OUTBOUND, navigate to the ‘Profile’ column and click on the icons shown there. Ensure that the Anti-spyware and URL Filtering profiles set.

53. To view the profiles, navigate to Panorama > Objects > Security Profiles > URL Filtering

54. The profile follows best practices recommendations for URL filtering.

55. Close and Commit the changes and push to device

If you see the message "Not all Commit jobs got triggered", see this section on how to resolve this issue.

56. Click in ‘Tasks’ button on the bottom horizontal bar  and ensure that ‘commit all’ job finishes. Click on Close to exit out of the window.

57. Relaunch the attack by navigating to the SSH session on att-app-server and issue the command.

/tmp/launch_attack.sh

58. This time we will see the ‘Hello, world!’ message but the attack does not succeed.
59. On the vul-app-server, you will see that the malware-sample file is not transferred. On the vul-app-server, execute these commands:

sudo docker exec -it vul-app-1 /bin/sh

ls -lrt /tmp

If you still see the malware-sample file here, please delete it and run the attack again.

60. Navigate to Panorama > Monitor > Traffic to see the firewall logging the web-browsing action. On the Filter search bar, paste the below query.

((addr.src in 10.1.1.100) and (addr.dst in 10.2.1.100) or (addr.src in 10.2.1.100) and (addr.dst in 10.1.1.100))

Deploy Zero Trust Policy to Limit Applications To and From Untrusted Sources

We can use the App-ID for ldap and rmi-iiop to block all RMI and LDAP to or from untrusted networks and unexpected sources.

61. Navigate to Panorama > Policies
62. Disable Policy 2 and enable Policy 3 ZT-LLDP-RMI_logdj_BLOCK

63. Commit the changes and push to device

If you see the message "Not all Commit jobs got triggered", see this section on how to resolve this issue.

64. Click in ‘Tasks’ button on the bottom horizontal bar  and ensure that ‘commit all’ job finishes. Click on Close to exit out of the window.

65. Relaunch the attack by navigating to the SSH session on att-app-server and issue the command.

/tmp/launch_attack.sh

66. Attack fails as the firewall block application access
67. Navigate to Panorama > Monitor > Traffic. On the Filter search bar, paste the below query.

((addr.src in 10.1.1.100) and (addr.dst in 10.2.1.100) or (addr.src in 10.2.1.100) and (addr.dst in 10.1.1.100))

Lab Teardown

To delete all the lab resources that were set up, you can run the below commands in CloudShell.

cd && lab-aws-zero-trust/teardown.sh

Note: Please note that this teardown script may fail due to insufficient privileges to delete all the resources. In such a case, you will need to once again update the IAM permissions for your user to get the privilege to delete resources, then run this script again.

Appendix

Connecting to the AWS Instances

There are 2 types of keys provided to connect to the AWS EC2 instances; PEM and PPK.

• If you are on a Mac system, you will be using ‘Terminal’ to connect to the devices via SSH. For this, click on the “Download PEM” link. This will download a file named “qwikLABS-L*****-*****.pem”.
• If you are using a Windows laptop to access this lab, you will need to have a SSH application like PuTTY installed. In this case, click on the “Download PPK” link. This will download a file named “qwikLABS-L*****-*****.ppk”.
• Make sure to note the location where the file is downloaded. On a Mac, by default, it should be downloaded to “/Users/<username>/Downloads”.

We will be using the user ‘ec2-user’ as the username to login to these applications. We can login to the EC2 instances using the below methods.

On the AWS CloudShell

• Navigate to the AWS CloudShell and run the below command to log in to the EC2 instance on the AWS environment. Make sure to replace the <instance-id> in the command below with the instance ID of the EC2 instance.

aws ec2-instance-connect ssh --instance-id <instance-id>

Using the PEM file

If connecting via the AWS EC2 console does not work, you could try the below two options as well.

• Launch terminal on your MAC.
• Navigate to the folder where the pem keys were downloaded to.

cd ~/Downloads

• Substitute qwikLABS-****.pem with the PEM key that you downloaded and run the command below

chmod 400 qwikLABS-****.pem

• Substitute qwikLABS-****.pem with the PEM key that you downloaded and run the command below. Username is ‘ec2-user’. IP Address is the public IP address of the ‘qwikLABS-vul-app-server’ that we noted in the previous step.

ssh -i qwikLABS-****.pem ec2-user@<vul-app-public-ip>

For example:

Using the PPK file

If you are using PuTTY to connect to the servers, after opening the putty,

• On the Category menu on the left, expand SSH and click on Auth.
• In the field to select the Private key file for authentication, browse for the downloaded PPK file and select it.

• Within the same PuTTY window, on the Category menu on the left, click on Session and provide the Public IP Address of the vul-app server as shown below.
• Click “Open” to connect to the server.

Troubleshooting

Setup script Errors

"Resource not found" or "No such File or Directory found" error

This error can occur due to come cloud resource, for example, transit gateway, gateway load balancer, etc., taking longer than required to be deployed, so the Terraform script times out. To resolve this error, simple run the setup command again.

Error: creating EC2 Instance: InsufficientInstanceCapacity

This error occurs due to exhaustion of AWS Cloud resources in the AWS region where we are deploying our lab. To overcome this issue, you can teardown your lab, change the AWS region and run the setup again.

Error: creating EC2 Instance: VcpuLimitExceeded

Error: creating EC2 Instance: VcpuLimitExceeded: You have requested more vCPU capacity than your current vCPU limit of 32 allows for the instance bucket that the specified instance type belongs to. Please visit http://aws.amazon.com/contact-us/ec2-request to request an adjustment to this limit.

This lab requires a total of 14 vCPUs, which is well below the limit of 32. This means that there already exists some AWS resources that were not cleaned up properly. There are a couple of ways to proceed from here.

• Teardown your lab. Then look for any EC2 instances that still exist in the AWS account and delete them. Run the setup again, or,
• Teardown your lab, change the AWS region and run the setup again, or,
• Teardown your lab, End the lab and Start the lab again from scratch.

Panorama Commit Issues

On selecting Commit and Push from the Commit menu on the Panorama, if you see the message "Not all Commit-All jobs got triggered", then

1. On the top right, click on Commit again and select ‘Push to Devices’

2. Click in ‘Tasks’ button on the bottom horizontal bar and ensure that ‘commit all’ job finishes. Click on Close to exit out of the window.

Activity Final Thoughts

As we conclude this session, you have familiarized yourself with building a policy model that provides multi-tier protection for your applications on AWS against Log4j, following best practices guidelines.