Setting Up Production Ready Infrastructure For Microservices
Service Fabric is Microsoft's answer to Microservices Orchestrator. It can help with Service Discovery, Fault Tolerance and containerizing your applications.
If you're using Service Fabric cluster to serve your production traffic, it's very important to create a secure cluster and set up reverse proxy to route traffic. I find myself creating Service Fabric clusters regularly for our clients. Instead of creating secure clusters manually everyone using the portal, I've created the Azure ARM template that I can use every time I need to create a cluster.
Here are the steps that I follow to deploy secure cluster in production for our customers.
- Create a Key Vault to store certificates and secrets
- Create a Virtual Network and configure Subnets
- Create Application Insights instance for logging telemetry and application events
- Set up HoneyComb for Observability
- Configure Azure Resource Manager Template for provisioning the cluster
- Deploy Traefik Reverse Proxy
- Configure Network Security Group rules to control traffic to the cluster
- Configure DNS rules in Cloudflare to route traffic to the cluster
- Restric traffic to the cluster only from the trusted IPs of Cloudflare
Creating a Key Vault
Azure Key Vault is a managed solution and provides an easy way to manage Keys, Certificates and Application Secrets. Developers have a tendency to store sensitive data like database connection strings, passwords etc., in the applications configuration files. While it's easy to get your applications up and running, storing sensitive data in a configuration file is a huge security risk.
No matter how elaborate your application architecture is, it's only as secure as it's weakest link. By storing sensitive data in configuration files, you're exposing the data to lot more people than the ones who need absolute access to. It's also a very bad idea to commit sensitive information to version control system as it can expose the data to everyone in your company.
There are various configuration stores available in the market to help you with abstracting configuration information away from your applications and only give role based access to the data. But they leave the responsibility of managing configuration server to developers. It adds yet another non-business critical application to the list to manage and monitor.
Azure Key Vault is a managed solution. You can provision a Key Vault for each environment at the click of a button and never worry about it going down. It exposes very rich API for creating and managing Keys, Certificates and Secrets. It's a must have for your Microservices Architecture.
Here's the list of commands you need to run to provision a key vault and upload keys and secrets.
Upload Keys
Connect-AzureRmAccount
Get-AzureRmSubscription
Set-AzureRmContent -SubscriptionId
New-AzureRmResourceGroup -Name 'techfabric-keyvault' -Location 'West US'
New-AzureRmKeyVault -VaultName 'TechfabricVault' -ResourceGroupName 'techfabric-keyvault' -Location 'West US' -SKU 'Premium'
The above set of commands helps you to connect to Azure Subscription, Create a Resource Group and provision a Key Vault in that Resource Group. Once the Key Vault is created, you can use the following command to upload keys in pfx file to Azure Key Vault
$PlainPassword = "verysecureplaintextpassword"
$SecurePassword = $PlainPassword | ConvertTo-SecureString -AsPlainText -Force
Add-AzureKeyVaultKey -VaultName 'techfabric-keyvault' -Name 'tecfabrickey' -KeyFilePath 'techfabric.pfx' -KeyFilePassword $SecurePassword
First your convert plain text password into a secure password and provide it as a parameter to the powershell command that upload keys in the pfx file to azure key vault.
Once this is done, the key is available for your application to use.
You can similarly upload the certificate too.
Import-AzureKeyVaultCertificate -VaultName "techfabric-keyvault" -Name "newcert" -FilePath "star.newcert.pfx" -Password $SecurePassword
Upload Secrets
Secrets are ideal for storing configuration information in a secure way. Here are the steps you need to do to upload secrets.
$PlainSecret = "VerySensitiveValue"
$Secret = $PlainSecret | ConvertTo-SecureString -AsPlainText -Force
Set-AzureKeyVaultSecret -VaultName 'techfabric-vault' -Name 'SecretPassword' -SecretValue $Secret
That's it. The secret is uploaded and available for you to use in your applications.
You can also read the secret value from Powershell commands
(get-azurekeyvaultsecret -vaultName 'techfabric-vault' -name "SecretPassword").SecretValueText
Provision a Secure Service Fabric Cluster
Creating a production ready service fabric application has been covered in my previous article here.
Provision an Application Insights Instance
Creating Application Insights Instance and configure your .NET Core Microservices to capture telemetry and other application events is covered in this article, here.
Set up HoneyComb for Observability and Testing in Production
Once your Microservices are deployed to production, it's important to observe how your applications are performing and be able to validate your assumptions. It's also very important to be able to take a peek on performance metrics, errors and other stats while it's being used live. HoneyComb enables just that.
Deploy Traefik Reverse Proxy to Service Fabric Cluster
Traefik is an awesome open source Reverse Proxy that protects your application from external threats and makes it very easy to provision SSL certificates, route traffic to the right services through Auto Discovery feature and helps with Tracing and Metrics.
We use Traefik in all our Service Fabric clusters and the process of deploying it as a guest executable into your service fabric cluster is covered in this article, here.
Once you deploy Traefik to Service Fabric, it automatically detects all the services running in the cluster and determines the routing rules. Traefik will be the only application in the Service Fabric cluster that'll be exposed to the outside world. Incoming traffic is evaluated and based on the routing rules configured in each service, the traffic is routed to appropriate service to handle the request. Since
Traefik acts a a middle-man, it can inspect the traffic and can make the decision to reject the request, or forward the request to the services. It can also automatically renew the SSL certificates using Let's Encrypt, so you never have to worry about expiring certificates in production.
Configure Inbound Rules in Network Security Group
It's not enough for you to set up a secure cluster. Each Subnet in a Virtual Network can be associated with a Network Security Group which has a set of Inbound and Outbound Rules.
It's very important to make sure all inbound traffic is denied by default. Then NSG rules can be applied to only those ports that needs to be exposed to outside world.
Cloudflare as Edge Proxy
Lastly, We recommend using Cloudflare as an Edge Proxy that acts as a front line of defense for all incoming traffic. Cloudflare is one of the worlds largest and most secure Edge Proxies. It has worlds leading technology to protect against DDoS attacks. Here's the list of all the things Cloudflare can do for you at the click of a button.
If there is a Denial of Service attack against your application, Cloudflare's technology will make sure those attacks are neutralized before any of that traffic reaches your applications. It can also help with proactively monitoring requests and rejecting anyone with malicious intent like SQL Injection attacks etc., Cloudflare can also help you with Edge Caching, saving you hundreds of giga bytes of bandwidth for requests that never reach your server.
Drawing from my experience of provisioning tons of Service Fabric Clusters for our clients, we've deployed many applications on Microservices architectures and have been very successful in mitigating any external threats and making sure the applications are secure and functioning to their full potential.