If, like me, you are running windows containers on an Azure Virtual Machine Scale Set and using RabbitMQ to communicate between containers and you want to scale up and down the VMSS based on queue size, then this article is for you!
Things you will need
- An Installation of RabbitMQ that has the API accessible
- Log in details for the RabbitMQ API
- An Azure subscription
- An Azure Service Principal that has access to your resources. (A bit of PowerShell below to help with that)
- Azure AD Tenant ID. (A bit of PowerShell below to help with that)
Use the below PowerShell to create a Service Principal called RabbitMQScale and set the role assignment. Just make sure you take note of the output as you will need this for some steps later on.
| $sp = New-AzADServicePrincipal -DisplayName RabbitMQScale | |
| $BSTR = [System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($sp.Secret) | |
| $UnsecureSecret = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto($BSTR) | |
| $AppID = (Get-AzADServicePrincipal -DisplayName RabbitMQScale).ApplicationID | |
| New-AzRoleAssignment -ApplicationId $AppID -RoleDefinitionName "Contributor" | |
| Write-host "Client ID =" $AppID | |
| Write-host "Client Secret =" $UnsecureSecret | |
| Write-host " Tenant =" (Get-AzContext).Tenant.Id |
Let’s get to it
So, in the Azure portal create a new Azure Logic App. You can click on All Services and then search for Logic Apps In the Logic apps blade just click on Add.
In here enter a name for your logic app. Select the subscription. Then either create a new Resource group or use an existing and select the location. Then if you like you can turn on log analytics so you can get some logs to query. Once finished click Create.
Now, once the logic app has been deployed in Azure click it so you can start to configure it.
You should now be in the Logic Apps Designer if you scroll down a tiny bit you will come across Blank Logic App click it.
Search for HTTP as this will be the first step. Click On the First option called HTTP under the triggers.
Under Method select GET. For URI enter the web address of your RabbitMQ API including the queue you want to monitor. So for example https://rabbitmqurl/api/queues/vhost/queuename.
Change authentication to Basic and then enter the Username and Password.
Change the Interval to whatever suits you. I have chosen 15 minutes.
Then click on New Step.
For this step search for Parse JSON and then click on the trigger with the same name.
Click on the Content box and then select Body in the popup window.
Now under the Schema bit, You can try generating one from a sample payload but you will end up needing to edit it. So below you will find the version I am using. It should work fine for you so feel free to just copy and paste into the box.
Once copied click on Next step.
Search for Read a resource and then click on the trigger called the same, but with (preview) at the end.
In here click on Connect with Service Principal. Bottom left.
In this box enter a Connection Name I just use the name of the service principal. Then enter the Client ID of the service principal and the Client Secret and Tenant id. If you used my PowerShell script from before you should have all of that. Then click Create.
In this new box. Select the subscription and Resource Group. For the Resource Provider select Microsoft.Compute.
For the Short Resource Id open a new tab and navigate to your VMSS that you want to scale and go to Properties.
In here copy the text under Resource ID after Microsoft.Compute.
Then paste that in the Short Resource Id box. If you have copied the first / just delete it.
Then in the Client Api Version box just enter 2019-07-01.
You should now have something like this.
Awesome, we now have the resource being read. Next, we need to create some variables. Click on Next step and search for Initialize variable and click the result.
Under Name type something like ADDSKU. Select Integer under Type. For the Value we are going to use an Expression. So click the Value box and then in the popup window select Expression at the top and paste the following in the fx box. Then click OK.
|
1 |
add(body('Read_a_resource')?['sku']?['capacity'],1) |
Next step is to add another variable but this time its to subtract from the VMSS rather than add. So go ahead and click Next Step again and search for Initialize variable again. As before select the result.
In the Name box enter MINUSSKU, change the Type to Integer and click the Value box again. In here click Expression and then paste the following into the fx box. Then click OK.
|
1 |
sub(body('Read_a_resource')?['sku']?['capacity'],1) |
The Magic bit
So now we are ready for the magic bit. Don’t worry it’s not really magic, it’s just a sort of nested condition.
Scale Up
So click Next step and search for Condition and select Condition in the results.
You should now see a nice Condition block with a nice if true and an if false box.
Under the Condition, box click on the Choose a value box. In the popup window search for messages and select it.
In the middle box set, the value to is Greater than or… Then in the last box enter the value that suits you. I am using 200.
Next, click Add. It’s just below where we were and select add row.
For this row we are going to set the maximum VM’s our scale set can scale to. I use 10. So under Choose a value select SKU Capacity.
Then in the middle box select is less than or eq… Then in the last box enter the max value you want your VMSS to scale to.
Under the true box click on add an action. Then search for create or update a resource and select it in the results.
Copy the same information you used for the Read resource step. You can expand the previous steps and just copy-paste the information.
Then click on the Add new parameter box. Put a tick in Location, SKU Capacity and SKU Name.
Select the Location of your VMSS. Then click on the SKU Name box and search for SKU Name. For SKU Capacity we are going to use one of the variables we created before. And as we are scaling up, we are going to use the ADDSKU one. Click the box and then click See more under Variables then click on ADDSKU.
That’s the scale-up bit sorted. Below you will see my finished if true block to make sure yours looks right.
Scale Down
Now in the if false block click on Add an action and search for Condition again and select it. In here we are again going to repeat the same as the scale-up but will change the middle and last boxes.
So, for the first row, Messages, select is less than or eq… In the middle box and then a number of messages in the queue, you are happy for the VMSS to scale down at. I use 50.
Now for the second row, SKU Capacity, select is greater than in the middle box and than a number for the lowest amount of VMs you want in your scale set. I use 2 to make sure I have some redundancy.
Under the new, if true section click on Add an action and search for Create or update a resource again.
Enter the same details that you have used previously and make sure you add the Location, SKU Name and SKU Capacity. But this time Under SKU Capacity select the MINUSSKU variable.
Now Click Save at the top left of the Logic App Designer blade.
The End?
Now after 15 minutes the logic app will run and will check your rabbit MQ queue to see if the VMSS needs to be scaled up or down. You can always add another action into the if true sections to send an email or teams notification to let you know that your VMSS has scaled.
You can also run the Logic app on-demand to help scale up or down quickly. Just go to the logic app blade and click run.
I have been using this for a while now and it seems to work really well for me. Before I was using CPU to scale and kept finding that some of the servers were not being utilised well.
If you have any questions or comments or a better way to do this let me know in the comments below or via social media.
0 Comments