Getting Started with Azure Automation

As a regular use of an MSDN account with Azure credits, I often find myself running out of credit before I expect and it’s often my own fault. It’s really easy to spin up a server or start experimenting with a service and forget to turn it off or delete it when done. If only a selection of my VMs would turn on during the day and off during the night…. Enter Azure Automation.

I looked at Azure Automation when it first came onto the scene, but admittedly struggled with getting it going. The certificate management related to the credentials was challenging, my PowerShell was less than stellar… and I just got frustrated and moved on to something else.

But after watching my subscription accidentally drain down to nothing 2 weeks before it reset, I had to revisit Azure Automation. The credential management is much easier now and the PowerShell required to start and stop VMs is minimal.

You’ll need two scripts… one to start and one to stop the desired VMs. You set a schedule for each one and sit back and relax. I’ve also been using GitHub a lot for source control and you can tie Azure Automation into that as well.

My scripts list out each VM I want on the nightly shut down schedule, because I didn’t want to have every VM in my subscription shut down automatically, in case I do have a need to keep something running for more than 24 hours. You may want to customize your scripts differently.

To get started, I recommend following the tutorial in the Azure documentation for “My First Runbook” . It walks you through all the steps of getting your automation account going as well as giving some sample scripts.

If you like to see the basic scripts I’m using for a few of my VMs, you can find them in my GitHub repo.

Automate Away!!

Deconstructing JSON: Counting On

So here I am, in JSON-land, working on my deployment for VMs. But so far, I’ve only deployed one VM in my network. But one VM is boring. I want to be able to rack and stack more than one machine at a time! 🙂  Enter the copy object and the copyIndex function!  You can read the official documentation about it, but let see how this works within my existing template. Things to note:

  • You can only reference the copyIndex function in RESOURCES. Don’t try to build variables with them, you’ll just be sad.
  • You can’t use a copy loop in a nested resource (this blog series hasn’t encountered this yet).
  • You can’t use them to loop items that are defined as properties within a resource, like multiple data disks.

So in my example, let’s extend the template to deploy 3 VMs in the same subnet and put them all in the same NSG (network security group). First, you must declare in the parameters that you’d like count something:

   "count": { 
      "type": "int", 
      "defaultValue": 3 
    },

Now, whenever you want to reference the “current iteration” of the loop elsewhere in your template, you’ll use copyIndex(). Another thing to note – computers like to count starting at zero, humans like to count starting at 1. If we stick with copyIndex(), we’ll get VMs numbered 0, 1, 2. If you’d like them to be numbered 1, 2, 3 use copyIndex(1), which will offset the numbering by 1.

Since this template is going to create 3 VMs, we will need to make sure to account for other resources that VMs have a one to one relationship with and be sure to loop those as well. The only items that we are required to loop are the network interfaces and the public IP addresses (which have individual DNS names).  We don’t have to loop things like the storage account (which can be used by multiple VMs) or the network, since we only need one in this case.

In the resource for the publicIPAddresses, you’ll see the reference to the copyIndex() in the name, and in the DNS domain label. In each case, the incremented number will be tacked on at the end of each. Then finally, you’ll see the specific reference to looping this resource added, with the name of “pubipcopy”, pulling the count from the parameter you set early.

     {
      "type": "Microsoft.Network/publicIPAddresses",
      "apiVersion": "2015-06-15",
      "name": "[concat(parameters('publicIPAddressName'), parameters('dnsNameForPublicIP'),copyIndex())]", 
      "location": "[parameters('location')]",
      "properties": {
        "publicIPAllocationMethod": "[variables('publicIPAddressType')]",
        "idleTimeoutInMinutes": 4,
        "dnsSettings": {
          "domainNameLabel": "[concat(parameters('dnsNameForPublicIP'),copyIndex())]"
        }
      },
       "copy": {
        "name": "pubipcopy",
        "count": "[parameters('count')]"
      }
     },

One of the reasons you need to name each resource copy object is so they can be used in places where dependencies are required.

For the network interface, you’ll see I’ve made similar references to the copyIndex() in the name. Also a network interface is dependent on the existence of a public IP address, so you’ll see “pubipcopy” referenced under “dependsOn” as well as in the IP configuration properties.

   {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "[concat(parameters('nicName'), copyIndex())]",
      "location": "[parameters('location')]",
    
      "dependsOn": [
        "pubipcopy",
        "[concat('Microsoft.Network/virtualNetworks/', parameters('vnetName'))]",
        "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]"
      ],
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
             "privateIPAllocationMethod": "Dynamic",
             "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses', concat(parameters('publicIPAddressName'),parameters('dnsNameForPublicIP'),copyIndex()))]"
              },
              "subnet": {
                "id": "[variables('subnetRef')]"
              }
              }
          }
        ],
       "dnsSettings": {
         "dnsServers": []
       },
       "enableIPForwarding": false,
       "networkSecurityGroup": {
               "id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]"
            }
      },
       "copy": {
        "name": "niccopy",
        "count": "[parameters('count')]"
      }
    },

This all comes together when you get to the VM resource where it refers to the NIC in the network profile and using the copyIndex() when creating the VM name, computer name and OS disk.

    {
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[concat('NanoVM-', copyIndex())]", 
      "location": "[parameters('location')]",
      "copy": {
        "name": "vmcopy",
        "count": "[parameters('count')]"
        },
      "dependsOn": [
        
        "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
        "niccopy"
      ],
      "properties": {
        "hardwareProfile": {
          "vmSize": "[parameters('vmSize')]"
        },
        "osProfile": {
          "computerName": "[concat('NanoVM-', copyIndex())]",
          "adminUsername": "[parameters('adminUsername')]",
          "adminPassword": "[parameters('adminPassword')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "[variables('imagePublisher')]",
            "offer": "[variables('imageOffer')]",
            "sku": "[parameters('windowsOSVersion')]",
            "version": "latest"
          },
          "osDisk": {
            "name": "[concat('osdisk_', concat('NanoVM-', copyIndex()))]" ,
            "vhd": {
              "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/NanoVM-',copyIndex(),uniqueString(resourceGroup().id),'-osdisk.vhd')]"
            },
            "caching": "ReadWrite",
            "createOption": "FromImage"
          }
          },
            
          "networkProfile": {
          "networkInterfaces": [             
        {
              "id": "[resourceId('Microsoft.Network/networkInterfaces',concat(parameters('nicName'), copyIndex()))]"
            }
          ]
        },
          "diagnosticsProfile": {
          "bootDiagnostics": {
             "enabled": "true",
             "storageUri": "[concat('https://',parameters('newStorageAccountName'),'.blob.core.windows.net')]"
          }
        }
        }

      }

You can find my full template here: https://github.com/techbunny/Templates/tree/master/multiple_nano_server

ARM Templates: What Error?!?

There is more than one way to deploy an ARM template once you’ve created or edited on to your liking. The first way is to copy the code into a blank template deployment in the Azure portal. This has a few benefits, primarily that it gives you the option to edit any parameters in the template to your liking before deployment and saves you from having to write a separate “azuredeploy.parameters.json” file.

However the deployment blade in the portal doesn’t really give you much in the way of error messages when the template fails. Often, you are give something along the lines of “error exists on line 1: position 10,374” or some such information. But your template is clearly more than one line in length and I have no intention of counting out thousands of characters to find the error location. The full text of the message may or may not be helpful depending on what point in the deployment the error occurred.

This is where deploying the template via PowerShell shines. As much as the red text is heartbreaking, it will give you the proper line number and position count to line up with your code. Also included is some often useable error text to point you on your way.

This single line of PowerShell will deploy a template that is hosted online (like in GitHub) and the -Verbose switch will report on the status of each of your resources as they start and complete deployment.

New-AzureRmResourceGroupDeployment -ResourceGroupName "<YourResourceGroupName>" -TemplateURI $templateFileURI -TemplateParameterURI $parameterFileURI -Verbose

If that doesn’t solve your problem or point you in the right direction, it’s time to dig deeper in the Azure Audit Logs from within the portal. They are chronological, so after a few clicks, you can drill down into the details of the most recent error. You can also access those logs via PowerShell:

Get-AzureRmLog -Status Failed -ResourceGroup "YourResourceGroupName" -DetailedOutput

You can also access these logs by digging down into the deployment in the portal. Just to “Audit Logs” and you’ll find all the activity in your subscriptions. Looked for the ones related to failed deployments or validations of deployments and as you drill into the details you’ll find what’s going wrong. Usually it’s an incorrect location of a parameters file, maybe a storage account name that’s already in use, or a missing resource group.

Happy Hunting!

How Microsoft IT does the Cloud

One of the things that is fascinating about working at Microsoft is watching how we function internally as our own “customer”. This is very apparent when it comes to integrating with cloud solutions like Azure. We have the same challenges as any other external customer, with the added bonus of developing, testing and implementing our own solutions. Sometimes people will ask me something like “You work for Microsoft, isn’t Azure just free for you?”

Alas, it is not. Every bit of Azure I use to learn, test and demo is accounted for somewhere, and I’m just lucky my manager hasn’t come after me holding an embarrassing bill to explain!

The MSIT Showcase has a host of documents, case studies and webinars to peel back the curtain and give you an idea of what it’s like to move many critical workloads into the cloud. Here are three links to get you started:

The cloud security mindset (Article) –
https://www.microsoft.com/itshowcase/Article/Content/703/The-cloud-security-mindset

Managing resource efficiency in a hybrid cloud (Webinar) –
https://www.microsoft.com/itshowcase/Article/Content/702/How-Microsoft-IT-manages-resource-efficiency-in-Azure-and-onpremises

Architecting your network for successful Azure adoption (Case study) –
https://www.microsoft.com/itshowcase/Article/Content/648/Architecting-your-network-for-successful-Azure-adoption

Enjoy!

A May Shower of Resources for IT Pros

It’s May (already!!) and with it brings a bunch of good resources for IT Pros looking to refresh their skills this spring.  Your first two stops should be the new IT Pro Cloud Essentials and IT Pro Career Center, which provide access to some great tools and features to help you tune up your skill set for the future, including:

  • A 3 month Pluralsight subscription
  • Role specific MVA training suggestions
  • One free exam voucher (limited supply, don’t wait to claim this!)
  • Azure credits and trials!
  • Guidance for career mapping, including salary information

If you are looking for something to read and want to refresh the items on your e-reader, this recommendation comes straight from Jeffrey Snover – the free e-book on Windows Server 2016 Technical Preview.

If you are already starting to architect a hybrid solution for your organization, I highly recommend checking out some of the architecture resources for hybrid cloud and other cloud related systems, like storage and networking.  These are great Visio diagrams and posters to help you understand and explain what changes might benefit your company.

  • Hybrid cloud overview  Microsoft’s cloud offerings (SaaS, Azure PaaS, and Azure IaaS) and their common elements.
  • Architecture of Microsoft hybrid cloud scenarios  An architectural diagram of hybrid cloud for Microsoft’s cloud offerings, showing the common layers of on-premises infrastructure, networking, and identity.
  • Hybrid cloud scenarios for Microsoft SaaS (Office 365)  The SaaS hybrid scenario architecture and descriptions of key hybrid configurations for Skype for Business, SharePoint Server, and Exchange Server (2 printed pages).
  • Hybrid cloud scenarios for Azure PaaS  The Azure PaaS hybrid scenario architecture and the description of an Azure PaaS hybrid application, with an example.
  • Hybrid cloud scenarios for Azure IaaS  The Azure IaaS hybrid scenario architecture and the description of a line of business (LOB) application hosted in Azure IaaS.

Finally, if you are a Bay Area local, keep an eye on the Microsoft Reactor events calendar.  The Reactor hosts a variety of community events on a variety of topics, including Microsoft and open source technology.

 

Deconstructing JSON: Adding a Network Security Group

If you’ve come from the Service Manager days of Azure, you’ll be familiar with the endpoints that came with each Cloud Service, handling the port forwarding needed to RDP to your server or to connect with PowerShell (port 5986). You could add or remove endpoints as need to provide access to whatever service you desired (after making the corresponding firewall change on the server itself.)

This is like parking your car in a garage.  You can choose to lock or unlock the doors (aka the OS firewall) and you can choose to provide access to the garage (aka Azure endpoints).

With ARM, when you deploy a server, you’ll be prompted to confirm the naming of your network security group, this network security group takes the place of the endpoints: providing some default rules, including the RDP access to port 3389.

So how does this differ from an endpoint?  One difference is that there is no port forwarding.  With a cloud service endpoint, the external port wasn’t 3389, because you could have multiple servers behind the single IP address assigned to the cloud service sharing the load balancer.   With ARM, the VM is assigned its own IP address without a load balancer, so the standard port number can be used. If you need to open an additional port, you simply add a new rule to the NSG and open the corresponding rule on the server firewall.

The second difference is that without a NSG (or the cloud service load balancer endpoints) to act as a gateway, your VM is basically just “hanging out there” with only its internal firewall to keep it safe.  So you really need to be sure you’ve checked that firewall configuration and you know what’s open to the public network.  By default, a Windows VM created with an Azure image has RDP and PowerShell/WinRM ports open on the operating system.

If you’ve been following along with my JSON template project, you’ll know that VMs can be created in ARM without the NSG.  And this fine if you are good with having your server parked out on the Internet with a few doors unlocked, so to speak.

But if you want to take advantage of the power of Network Security Groups to limit the sources of connections or to customize the way traffic flows between tiers of servers on your network, you’ll need to include the NSG resource in your template.

First you’ll  need to add a parameter to declare the Network Security Group.

  "networkSecurityGroupName": {
            "defaultValue": "NanoVMnsg",
            "type": "string"
    },

Then you’ll to configure the resource itself to include the security rules.

{
      "type": "Microsoft.Network/networkSecurityGroups",
      "name": "[parameters('networkSecurityGroupName')]",
      "apiVersion": "2015-06-15",
      "location": "[parameters('location')]",
      "properties": {
            "securityRules": [
                {
                    "name": "winrm",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "5985-5986",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 110,
                        "direction": "Inbound"
                    }
                  },
                  {
                    "name": "rdp",
                    "properties": {
                        "protocol": "Tcp",
                        "sourcePortRange": "*",
                        "destinationPortRange": "3389",
                        "sourceAddressPrefix": "*",
                        "destinationAddressPrefix": "*",
                        "access": "Allow",
                        "priority": 120,
                        "direction": "Inbound"
                    }
                 }
                ]
            },
            "dependsOn": []
        },

Finally, you’ll need to adjust your network interface resource to depend on the NSG.

{
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "[variables('nicName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]",
        "[concat('Microsoft.Network/virtualNetworks/', parameters('vnetName'))]",
        "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]"
      ],
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
             "privateIPAllocationMethod": "Dynamic",
             "publicIPAddress": {
                "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
              },
              "subnet": {
                "id": "[variables('subnetRef')]"
              }
              }
          }
        ],
       "dnsSettings": {
         "dnsServers": []
       },
       "enableIPForwarding": false,
       "networkSecurityGroup": {
               "id": "[resourceId('Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]"
            }
      }
    },

Go to my GitHub repository to find the full version of this template. By default, it will deploy a Nano Server VM.

Now here’s a little gotcha when it comes to NSGs and remote server management (as of this writing): If you created a VM from the Azure Portal and accept all the defaults (which include an NSG), that NSG doesn’t open the ports for WinRM by default.  It only opens RDP.  The OS firewall is open to accept WinRM and PowerShell, but the NSG blocks it.  You need to edit the NSG to include TCP ports 5985 (http) and/or 5986 (https) for remote management.

You might not notice this issue if you are connecting to full GUI servers with RDP, but if you are trying out Nano Server there is no RDP, so you must be able to connect with PowerShell.

Deconstructing JSON: Anatomy of a VM

While the Azure VM on has two official dependencies, the storage account and the network interface, there are a variety of properties that you need to gather or determine ahead of time in order for your deployment to be successful.  If you were doing the manual creation of the VM in the Portal, the properties would be the parts you’d have to enter while shuffling through all the blades.  Let’s shuffle less blades….

With the VM template, we start out like every other resource, requiring a type, name, API version and location for deployment.  You can see the two lines that call the two dependencies.

{
      "apiVersion": "2015-06-15",
      "type": "Microsoft.Compute/virtualMachines",
      "name": "[variables('vmName')]",
      "location": "[parameters('location')]",
      "dependsOn": [
        "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
        "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
      ],

When you get into the properties, you can see where everything comes together with a variety of profiles for hardware, the operating system, storage, networking and diagnostics.  When deploying via PowerShell, you’ll get prompted to enter the Admin username and password manually.

To fill in the variables for the image reference part of the storage profile, you can use some handy PowerShell to query Azure for the existing image names.  (I’ll put that at the bottom of the post.)

      "properties": {
        "hardwareProfile": {
          "vmSize": "[variables('vmSize')]"
        },
        "osProfile": {
          "computerName": "[variables('vmName')]",
          "adminUsername": "[parameters('adminUsername')]",
          "adminPassword": "[parameters('adminPassword')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "[variables('imagePublisher')]",
            "offer": "[variables('imageOffer')]",
            "sku": "[parameters('windowsOSVersion')]",
            "version": "latest"
          },
          "osDisk": {
            "name": "osdisk",
            "vhd": {
              "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
            },
            "caching": "ReadWrite",
            "createOption": "FromImage"
          }
          },
            
          "networkProfile": {
          "networkInterfaces": [             
        {
              "id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
            }
          ]
        },
          "diagnosticsProfile": {
          "bootDiagnostics": {
             "enabled": "true",
             "storageUri": "[concat('https://',parameters('newStorageAccountName'),'.blob.core.windows.net')]"
          }
        }
        }

The PowerShell for the image references: these lines will return the current images available for Windows Server. Be sure to pick the location you intend to use, because some image skus have limited regions and change the publisher name as needed if Windows Server isn’t what you are looking to deploy.

Get-AzureRmVMImagePublisher -Location "West US"

Get-AzureRmVMImageOffer -Location "West US" -PublisherName MicrosoftWindowsServer

Get-AzureRmVMImageSku -Location "West US" -PublisherName MicrosoftWindowsServer -Offer WindowsServer

You’ll find the full template to deploy this one VM here: https://github.com/techbunny/Templates/tree/master/smallcloudlab