Here is my full study guide in entirety for download. Feel free to let me know if there is something that needs correction etc. Hope this helps. Cheers!
Section 7 – Administrative and Operational Tasks in a VMware vSphere Solution
Objective 7.1 – Manage virtual networking
I’ve gone over virtual networking a bit already. But there are two basic types of switches to manage in vSphere. Virtual Standard Switches and Virtual Distributed Switches. They both have the same components. Virtual Ports Groups, VMkernel Ports, and Uplink Ports. Here is a diagram depicting how it might look on a host
VMkernel ports are used for management purposes. When you set it up, you can choose using it for the following purposes
- vMotion – this is used to migrate VMs
- Provisioning – used for VM cold migration, cloning, and snapshot migration.
- Fault Tolerance logging – enables Fault Tolerance logging on the host (you can only have one per host)
- Management – management communication between hosts (should have minimum of two for redundancy)
- vSphere Replication – Handles outgoing replication data sent to the vSphere Replication Server
- vSphere Replication NFC – Handles incoming replication data on the target replication site.
- vSAN – allows for vSAN traffic, every host that is part of a vSAN cluster must have one.
VM Port Groups are for VM network traffic. Each of the VMs have a virtual NIC which will be part of a VM port group.
Uplink ports are connected to physical NICs. A Virtual Distributed Switch will have an uplink port group that physical NICs from multiple hosts.
You can manage your networking from a few locations in the HTML5 client. You can also manage hosts from the host HTML5 client. In the HTML5 client you manage networking from Host > Configure > Networking shown here.
You can then change manage the components as needed. If you need to manage a Virtual Distributed Switch you can do that there as well or you can create a VDS on the networking tab in the navigation pane.
You can configure shares and other settings here as well as you can see. You can find more info here if needed.
There is also managing the virtual networking of the VM. If you right click on the VM and then select Edit Settings. You can edit the networking adapter type and what virtual network the VM is connected to.
You can also migrate multiple VMs to another network if you go to the network tab in the navigation pane. Clicking the following will pop up a wizard.
In the wizard you select the destination network.
Then you select all the VMs you want to migrate.
Then you complete it.
Objective 7.2 – Manage datastores
Datastores are logical storage units that can use disk space on one disk or span several. There are multiple types of datastores:
To manage them, you can navigate to the Datastores tab on the navigation pane and select the datastore you want to manage. Then click on Configure on the object pane in the middle.
From this screen you can increase the capacity. Enable SIOC, and edit Space Reclamation priority. Using the Connectivity and Multipathing, you can edit what hosts have access to this datastore. You can also see what files and VMs are on this datastore. You can perform basic file functions through this as well.
To dig a little deeper though. How did we get here? How do we see the original device? To do that we have to go back to the host configuration. There we look at two main things. Storage Adapters and Storage Devices
This will show us what our host is able to get to. If we don’t have access to something we may need to either add it if it’s ISCSI or NFS or Protocol Endpoint if its a vVOL. Once we can see the RAW device or we have finished setting up the share or protocol endpoint, we can right click on a host and select Storage > New Datastore. This pops up a wizard that looks like this
The next screen will allow us to give the datastore a name and what device we want to use for it. Then we choose a VMFS version. We would choose 5 if we still had older hosts running older vSphere. We would choose 6 if we had all 6.5 or 6.7. Why would you want to use it? Look here for a nice table. You can then partition it if desired and finish.
Objective 7.3 – Configure a storage policy
- To create a storage policy, click on the Menu drop down at the top of your HTML5 client and choose Policies and Profiles
- Click on VM Storage Policies
- Select Create VM Storage Policy and on the popup wizard, give it a name.
- This screen allows you to choose between Host Based Services or Datastore Specific rules. Host based are specific services that particular host may provide such as caching, encryption, etc. These can be used in conjunction with Datastore specific rules which are directed to specific datastores. Such as I tag a specific datastore as “Gold” storage and I create a Storage policy that requires a VM to use “Gold” storage. I am going to use the tag-based placement option.
- I have already created a Tag category called Storage Type and I am going to tell it to Use storage tagged with the “Gold” tag. I could tell it to not use that tag as well. Multiple Rules can be used at the same time.
- I have one Datastore tagged as “Gold” Storage.
- That’s it. Click Finish and you have created a Storage Policy. Just to show you what host based services might look like here is a screenshot
Objective 7.4 – Configure host security
There are several built-in features that can secure a host. Let’s go over them
- Lockdown Mode – When enabled this prevents users from logging directly into the host. It will only be accessible through the local console if you are on an accepted user list or vCenter. You can also turn off the Direct Console UI completely. This can be found under Configure > Security Profile
- Host Image Profile Acceptance Level – This is like driver signing on a Microsoft Windows machine. This will only allow bundles or drivers with an acceptance level you set.
- Host Encryption Mode – This setting encrypts any core dumps from the host.
- Firewall – There is a stateless firewall included in ESXi. Most ports are locked by default. If you want to add a new port not already in the list you will need to do it at command line.
Objective 7.5 – Configure role-based user management
Role-based management allows you to assign a set of permissions to a user or group. This is great as this makes it easier to assign just the permissions you need to a user and no more. This is great for security. VMware provides a number of Roles pre-configured. These can’t be changed. What you can do, is clone them and change the clones. You can also create your own custom role. In order to do this, you click on the Menu and go to Administration
You can see the predefined roles when you select Roles under Access Control
To clone you select one and then click the Clone icon
You need to name it and click ok on the window the pops up. To edit the clone you just made, click on the Pencil icon after selecting the new role. Then select the privileges you want to allow or disallow by clicking on the check boxes.
You can see the privileges already assigned to a role by clicking on the Privileges button on the side.
You then assign the roles under the Global Permissions item. You can use one of the built-in user or groups or you can add a new user/group. You can add the group from any of the Identity sources you have setup already.
When you add or edit the permissions you set the role.
There is a special role called No Access as well that you can assign to a user to keep them from accessing specific objects or privileges.
Objective 7.6 – Configure and use vSphere Compute and Storage cluster options
After you create a cluster, you can right click on it and select settings, or click on the configure tab in the center, object pane
Quickly going through the options available. There is DRS and HA we’ve already gone over. We then have:
- QuickStart – is a wizard to help you configure your cluster.
- General – lets you change the swap file location for your VMs. This will be the default setting for the cluster. Default VM compatibility is the default VM Hardware version for the cluster.
- Licensing – This is only used if you vSAN
- VMware EVC – This was mentioned previously as well. Enhanced vMotion Compatibility. This allows you to use disparate versions of processors and vMotion between them.
- VM/Host Groups – This is the VM Groups and Host groups you can setup to create Affinity or Anti-Affinity rules
- VM Host Rules – These are the Affinity or Anti-Affinity rules.
- VM Overrides – This allows you to override cluster settings for DRS/HA restart or response for individual VMs.
- Host Options – Allows for host power management. You enter in your IPMI settings per Server
- Host Profile – This will be gone over in a few objectives, but creates a settings template for all hosts in the cluster.
- I/O filters – You can install I/O filters here (VAIO) This can be a plugin such as backup or disaster recovery filters.
- Alarm Definitions – This is where you can add/enable/disable/delete alarms for your cluster (applies to objects in the cluster)
- Scheduled Tasks – You can schedule certain tasks for off hours. New Virtual Machine, Add Host, or Edit DRS.
- vSAN – This won’t say much here unless it’s turned on.
A Datastore Cluster or Storage Cluster (unless referring to VSAN cluster) is created by right-clicking on the datacenter in the Storage heading on the object pane.
- This launches a wizard to go through. You will need to enter a Datastore Cluster name and you should turn on Storage DRS
- You then are presented with more options than anyone should be. The first is what level of automation would you like, but then you have all these other options which I will leave at cluster default. Each one of them will check certain metrics or alarms and move the VM storage based on what it sees.
- Now you need to decide storage DRS runtime settings. These are thresholds you set before it takes action to move data around. I’m leaving defaults again.
- You then select your cluster and / or hosts that will participate in sharing their datastores in this.
- Select the datastores that will make up this Datastore cluster
- It gives you final summary screen and you click Finish.
Objective 7.7 – Perform different types of migrations
We’ve already gone over the types of migrations possible. Now let’s see how to accomplish them.
- To migrate a VM, whether you migrate the VM or storage, you need to right click on the VM and choose Migrate.
- You are given the option of 3 types of migration. vMotion = Compute resource only, svMotion = Change storage only, or enhanced or xvMotion is both. The screens after depend on which you choose here. I will choose both so you see both screens.
- For the compute resource to migrate to, I need to choose either a cluster, or individual host. A handy little tidbit that’s nice is the upper right-hand corner. VM origin tells you where this VM is sitting right now, both host and datastore.
- Select storage next.
- Next, select the network for this VM to use.
- vSphere gives a summary, click Finish and it will migrate.
Objective 7.8 – Manage resources of a vSphere environment
There are several resources that can be managed in a vSphere environment. There are mechanisms built-in to vSphere to allow that. You can create resource pools, assign shares for CPU, memory, disk, and network resources. You can also create reservations and limits. Let’s define a few of those and how they work.
- Reservations – this is the amount of the resource that is guaranteed. If the resource can’t be given, the VM will not power on.
- Limits – are the maximum amount of that resource you will allow for that VM. The issue with limits is if you have extra resources vSphere will still not allow that VM to have more resources.
- Shares are used to compete for the resources between. Shares will only come into play when there is contention for it. During regular periods when all the VMs are happy and there is plenty of resources, shares don’t matter.
Resource Pools can also be created to slice off resources. You can have reservations on Resource Pools as well, but you can do a bit more. You can have expandable reservations to borrow resources from its parent if it needs to. This is what you need to configure when you create a CPU and Memory Resource Pool
You can also assign this on an individual VM basis
To assign disk shares you can look at the individual VM
You can also assign shares and manage network resources on Virtual Distributed Switches with Network I/O Control enabled.
Objective 7.9 – Create and manage VMs using different methods
There are several methods to create VMs. You can:
You can also deploy from an OVF template, use the OVF Tool or create a VM from a physical using the P2V tool. For the purposes of the exam they more than likely just want you to know about the ones in the picture and deploying from an OVF template.
You can manage VMs through the HTML5 client, API, PowerCLI (PowerShell) or even through the ESXi host console. There are even some options you can only do using PowerCLI. Creating a new VM via PowerCLI isn’t hard either, it can be done with command like the following:
New-VM -Name ‘TestVM’ –VMHost ‘VMHost-1’ -Datastore ‘TestDatastore’ -DiskGB 40 -MemoryGB 8 -NumCpu 2 -NetworkName ‘Virtual Machine Network’
That creates a new VM with the name TestVM on VMHost-1 storing its 40GB VMDK on the TestDatastore. A lot simpler than going through a long wizard to me.
Objective 7.10 – Create and manage templates
Templates are VMs that have been converted so that they can’t be turned on. They are used as base server machines or VDI base workstations. Creating them is a simple process. You can do this with a running VM by cloning it (creating a copy) and making the copy a Template. If you want to convert the machine you are working on, it will need to be turned off. I will go over both ways to do this.
- Right click on the VM to be converted. We will start with a running VM.
- Give the VM Template a name
- Choose a location for the template
- Choose storage for the template
- Complete by clicking Finish.
For a machine that is turned off you can clone it as well, but you have the option of turning that VM into a template. To do that:
- Right click on the VM you want to change to a template.
- If you choose Convert to template, it asks you if you are sure and then does it. If you Export OVF this will save an OVF file to your desktop that is the VM in template format that you can import like an appliance.
Objective 7.11 – Manage different VMware vCenter Server objects
I’ve gone over how to manage different types of objects so I will take a stab here and guess that they are referring to the actual vCenter Server objects and not clusters, hosts, etc.
To manage the vCenter Server object, there is a couple of places to go to. The first is Administration > System Configuration. This location will allow you to export a support bundle, converge an external PSC to embedded, and decommission PSC. Oh, you can also reboot it.
The next place you can configure the vCenter is by clicking on the vCenter in the navigation pane and then go to the configure tab in the object pane. You can see that here
This is just changing the settings on the vCenter server itself and not the object.
If anyone has a thought on what they may be looking here if I didn’t cover it, reach out to me.
Objective 7.12 – Setup permissions on datastores, clusters, vCenter, and hosts
Permissions can be set on most objects in the vSphere environment. To do that you need to navigate to the Permissions tab in the object pane. Here is an example
You can see how you can assign permissions to it. Click on the ‘+’ in order to add another user or group to it. You can also edit an existing permission by clicking on the pencil icon. You can also propagate this permission to its children with the Propagate to children checkbox.
If a user has conflicting permissions, the explicit permissions will win over general. This allows you to assign a user “No Access” to an object and it will win over having group rights to it. The user documentation has this really well. (From the VMware Documentation here)
If multiple group permissions are defined on the same object and a user belongs to two or more of those groups, two situations are possible:
No permission for the user is defined directly on the object. In that case, the user has the privileges that the groups have on that object.
A permission for the user is defined directly on the object. In that case, the user’s permission takes precedence over all group permissions.
Objective 7.13 – Identify and interpret affinity/anti affinity rules
Affinity and Anti-Affinity rules exist on a DRS enabled cluster. They are typically used for the following reasons:
- Affinity Rules – Used for multi-tier app VMs or other VMs that communicate heavily or depend on each other in order to run. It can also be used to keep a VM running on a specific host for licensing or other purposes.
- Anti-Affinity Rules – Use to keep VMs separate from each other or keep them from running on separate hosts.
These rules can be setup as “Must” rules or “Should” rules. Just like it sounds the Must will prevent the machines from doing what is instructed and if they can’t comply with the rule they won’t start. The Should rules will try everything they can to comply but for example, you are down to one host, the machines will still run there as that is their only option.
You create groups that are made up of either VMs or hosts and then create a rule that defines the relationship between them. You set them up underneath the Configure tab under your cluster. Here is what that looks like:
You would create the VM and/or host groups. Then you create the rules that will govern them.
Objective 7.14 – Understand use cases for alarms
Use cases for alarms are plentiful. You don’t want errors and issues happening in the background without you knowing. Even better, it would be great to get notice of these events before they happen. That is what alarms can do for you. They can notify you in response to events or conditions that occur to objects in your vSphere environment. There are default alarms setup for hosts and virtual machines already existing for you. You can also setup alarms for many objects. An alarm requires a trigger. This can be one of two things.
- Condition or State. This is monitoring the condition or state of an object. And example of this would be a datastore is using 80 percent of its storage. Or a host is experiencing high CPU usage.
- Event. This would be something like a host hardware changes, or leaves a cluster.
You can setup an alarm by right clicking on the object and then click on Alarms > New Alarm Definition.
Objective 7.15 – Utilize VMware vSphere Update Manager (VUM)
VUM (vSphere Update Manager) is VMware’s server and management utility to patch and upgrade its software. While there were many requirements to get VUM working on previous versions of vSphere, in 6.7 its pretty easy. Though its not completely simple, it does make more sense once you use it for a little bit. First, we need to define a few terms.
Baseline – is one or more patches, extension or upgrade that you want to apply to your vSphere Infrastructure. You can have dynamic patches or static. Dynamic baselines will automatically download and add new patches. I don’t necessarily recommend this as you don’t know how a patch will affect your environment without testing. Now if it’s a test environment go for it! VMware includes two dynamic baselines for patches predefined for you. You can create your own.
Baseline Group – Includes multiple baselines. The pre-defined ones are Non-Critical and Critical Patches. Unless one causes an issue, it would be good to have both of those. I created a group that includes both called Baseline Group 1.
You can create a baseline that includes an upgrade say from 6.5 to 6.7 as well. There are settings that go along with this service and here is what they look like.
- Patch Downloads concerns itself with getting your updates.
- Patch Setup concerns itself with where it is getting them from. Do you need a proxy?
- Recall Notification. Occasionally VMware needs to recall a patch that isn’t up to par. This setting will notify you there is a recall and what it is and make sure it doesn’t apply that patch to any hosts.
- Network Connectivity. Connectivity for VUM. Mainly port numbers and host name.
- Patch Downloads concerns itself with getting your updates.
- Hosts – When you apply the baselines to a host, what do you want it with the VMs, host if it uses PXE to boot, and retries.
- VMs – If you are remediating VMs do you want to take a snapshot automatically and how long do you want to keep them.
- Hosts – When you apply the baselines to a host, what do you want it with the VMs, host if it uses PXE to boot, and retries.
The setup of the server is just the first step though. You now need to get these patches to the hosts and VMs. You have two options when you apply them. You can Stage, or Remediate. Stage will just load the patches on it and wait for you to tell it to take action. Remediate takes immediate action. You can do this by going to the update tab for the object. Here is the update for the cluster.
At the bottom you notice I attached the baseline. This is needed to stage or remediate your hosts and VMs. You can then check them by Checking Compliance. You may also notice you can update VMware Tools and VM Hardware versions en masse. (may require VM reboot)
Objective 7.16 – Configure and manage host profiles
Host profiles provide a mechanism to automate and create a base template for your hosts. Using host profiles, you can make all your hosts exactly the same. VMware will inform you if your host is not in compliance yet and then you can take steps to remediate it.
You access it under Policies and Profiles
There is a process to it. Here it is:
- Click on Host Profiles on the navigation pane on the left.
- Next is Extract Host Profile. This is going to be taking a host you select and that will be the “baseline”
- This will pop up a wizard. This is where you select the host.
- Give it a name and a description and then Finish
- Once that is done, you now have a window that looks like this
- Yes, its small. The point is when you click on the host profile you now have additional options above. Notice as well that the profile is also a hyperlink. Click on it.
- Click on the Actions to attach to hosts or clusters.
So that is the end of this study guide. If you find something incorrect in it or I didn’t understand the Blueprint from VMware, let me know. I appreciate you taking the time to read through and hope you were able to use it. I really appreciate the community and all the things its done for me, which is why I love doing things like this. Thanks!!
Mike Wilson (IT-Muscle.com / @IT_Muscle )
Section 5 – Performance-tuning and Optimizing a VMware vSphere Solution
Objective 5.1 – Determine effective snapshot use cases
Many companies use the term snapshot. There are numerous definitions for snapshots that vary on the company. We should first define what VMware does with snapshots.
VMware preserves a Point in Time or PIT for a VM. This process freezes the original virtual disk and creates a new Delta disk. All I/O is now routed to the Delta disk. If data is needed that still exists on the original disk it will need to go back to that to retrieve data. So now you are accessing two disks. Over time you can potentially double the size of the original disk as you make changes and new I/O. The original 10 GB disk becomes 20 GB over 2 disks. If you create more snapshots, you create new Delta disks and it continues.
Now that we understand a bit more about them, we see the limitations inherent. This tool was never meant to be a backup. It was designed to be used for reverting back to the original (if needed) after small changes. Most backup tools DO use snapshots as part of their process, but it is only used for the amount of time needed to copy the data off and then the snapshot is consolidated back again. Here are a few Best Practices from VMware on how to use them.
- Don’t use snapshots as backups – major performance degradation can occur and I have seen people lose months of data or more when the chain got too long.
- 32 snapshots are supported, but it’s better not to test this.
- Don’t use a snapshot longer than 72 hrs.
- Ensure if you are using a 3rd Party backup that utilizes the snapshot mechanism, they are getting consolidated and removed after the backup is done. This may need to be checked via CLI
- Don’t attempt to increase disk size if the machine has a snapshot. You risk corrupting your snapshot and possible data loss.
Most use cases involve you changing the VM or upgrading and once you find out it does or doesn’t work, you remove the snapshot. A good example of this is Microsoft Windows Updates. Create a snapshot, install the updates and test. If the updates haven’t broken anything, consolidate. Another use case might be installation or upgrade of an important program. Or a Dev use case of changing code and executing to determine if it works. The common thread between all the use cases is temporariness. These use cases are for snapshots running a very short period of time.
Storytime. I had a company that called in once that was creating snapshots for their Microsoft Exchange Server. They were taking one every day and using it as a backup. When I was called, they were at about a year of snapshots. Their server wasn’t turning on and trying to remove the snapshots wasn’t working. Consolidation takes time and a bit more space. We tried to consolidate but you can only merge 32 snapshots at a time. They got impatient about 25% through the process and tried to turn it on again. When that didn’t work, they had to restore from tape backup and lost a decent amount of data.
Objective 5.2 – Monitor resources of VCSA in a vSphere environment
Monitoring resources can be done from more than one place. The first place is in the vCenter appliance management page at :5480. After you log into it, you have the option on the navigation pane called Monitor. This is what it looks like:
Notice the subheadings. You can monitor CPU and Memory, Disks, Networking, and the database. You can change the time period to include metrics up to the last year. Since the VCSA is also a VM, you can view this from inside the vSphere HTML5 client. This view allows you to get a bit more granular. You are looking at it from the hosts perspective whereas the Appliance Management page is from within the VM. Both are important places to give you a full look at how the vCenter is performing. Here is a screenshot of inside the HTML5 client of my vCenter Appliance.
You can attach to the vCenter via SSH or console and run TOP for a per process view of the appliance. Here is what that looks like
These are the most common ways you would monitor resources of your VCSA.
Objective 5.3 – Identify impacts of VM configurations
There much to unpack with this objective. I will work through best practices and try to stay brief.
- While you want to allocate the resources that your VM needs to perform, you don’t want to over-allocate as this can actually perform worse. Make sure there are still enough resources for ESXi itself as well.
- Unused or unnecessary hardware on VM can affect performance of both the host and all VMs on it.
- As mentioned above, over allocation of vCPU and memory resources will not necessarily increase performance and it might lower it.
- For most workloads, hyperthreading will increase performance. Hyperthreading is like a person trying to eat food. You have one mouth to consume the food, but if you are only using one arm to put the food in, it isn’t as fast as it could be. If you use both arms (enabling hyperthreading) you still only have one mouth (one core) but you aren’t waiting for more food and just keep constantly chewing. Certain workloads that keep CPU utilization high, benefit less from hyperthreading.
- Be aware of NUMA (Non-Uniform Memory Access). Memory is “owned” by sockets. If you use more memory than that socket owns, you need to use memory from the other socket (if available). This causes a small delay because it has to move across the bus vs right next to the processor. This can add up. (Oversimplified but the idea is there). There are policies that can be set that could help if needed. Not in the scope of this certification though.
- Not having enough physical memory can cause VMs to use slower methods of memory reclamation all the way to disk caching. This causes performance degradation.
- Creating shares and limits on your machine may not have the result you believe. Weigh those options carefully before you apply them.
- Make sure you use VM Tools in your VMs as they add a number of useful and performance increasing solutions.
- The hardware you use in the configuration can also change performance. For example, using PVSCSI vs LSI SAS or using VMXNET vs E1000 NICs can make a decent performance jump.
- Make sure you use VMware snapshots how they were intended and not for long periods of time.
- There are different types of VMDKS you can create. They include thin provisioned, thick (lazy zeroed), and thick (eager zeroed). There are reasons you might utilize them. Thin disks are the best in a scenario where you may not have all the space yet. You may need to buy more disks or they may be already on their way. Eventually you will have this space. It is important that you monitor your space to make sure you don’t consume it before you have it. If you do, the VM will be suspended best case, worst case you can lose data. Thick (lazy zero) is when you fence all that space off for that disk up front. You can’t over-provision this, you have to already have the disk space. The “lazy zero” comes in play when you go to use the space. VMware will need to format the disk block before using it. This can potentially be a slow down if there are a high number of writes to the disk. If the VM is more read heavy, you are just fine. Thick (eager zero) will take more time to create, because it formats the whole disk up front before use. This type if best for a VM with heavy writes and reads such as a DB server etc.
Keep these in mind when creating VMs and also take a look at the VMware Performance Best Practices guide here.
Section 4 – Installing, Configuring, and Setting Up a VMware vSphere Solution
Objective 4.1 – Understand basic log output from vSphere products
VMware has come a long way from when I started troubleshooting their products. Their logs have gotten easier to get to, and improved in their quality. What I will do here is give you a quick overview of where to find the logs and how to read them.
Where before the easiest option was to open a SSH session to the host and look at the logs, you can easily do that from within the host UI now. If you go to Monitor you can see a list of all the logs available to peruse.
Here in the screenshot, you can see
- Monitor menu and the tab for logs
- Logs available
- Log output
And here is a list of the logs on the ESXi host along with a description for what the log keeps track of.
You can still access these logs through the DCUI or a SSH session as well.
Alright so you got the log now… How do you use it? Here is a sample taken from a VMKernel.log. This was after shutting down a switch port using a Software ISCSI controller to a SAN LUN.
2013-12-05T21:42:47.944Z cpu25:8753)<3>bnx2x 0000:04:00.0: vmnic4: NIC Link is Down
2013-12-05T21:43:12.090Z cpu16:8885)WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhba45:CH:0 T:0 CN:0: iSCSI connection is being marked “OFFLINE” (Event:4)
2013-12-05T21:43:12.090Z cpu16:8885)WARNING: iscsi_vmk: iscsivmk_StopConnection: Sess [ISID: 00023d000001 TARGET: iqn.2001-05.com.equallogic:0-8a0906-0f6407f09-1173c8a93ab4f0f6-aim-2tb-1 TPGT: 1 TSIH: 0]
2013-12-05T21:43:12.090Z cpu16:8885)WARNING: iscsi_vmk: iscsivmk_StopConnection: Conn [CID: 0 L: 192.168.3.123:61632 R: 192.168.3.3:3260]
2013-12-05T21:43:22.093Z cpu31:8261)StorageApdHandler: 248: APD Timer started for ident [naa.6090a098f007640ff6f0b43aa9c87311]
2013-12-05T21:43:22.093Z cpu31:8261)StorageApdHandler: 395: Device or filesystem with identifier [naa.6090a098f007640ff6f0b43aa9c87311] has entered the All Paths Down state.
Let’s decipher this a bit more.
- This part is the time stamp of the log entry.
- This is what is the reporter. In this case it is the bn2x driver
- This is what it is reporting on, specifically vmnic4 at the hardware address referenced 0000:04:00:0
- This is data about what it saw. Namely the NIC link went down.
Some entries are a bit more difficult to read than others but the structure stays pretty close. You can also use something like Log Insight to help search through the logs and decipher them.
vCenter Server Logs
We have logs we may need to retrieve for vCenter Server as well. Unfortunately, it doesn’t have a browser like the hosts. (Hint Hint VMware) Here is where you can get to them though.
This is accessing the Appliance Config at port 5480.
Once this is done downloading you have a decent size .tar file. You will need to unzip this a couple times. When you finally have a regular directory structure all the logs will be under the /var/log/vmware folder. Here is a list of the files and locations and what they do.
|Windows vCenter Server||vCenter Server Appliance||Description|
|vmware-vpx\vpxd.log||vpxd/vpxd.log||The main vCenter Serverlog|
|vmware-vpx\vpxd-profiler.log||vpxd/vpxd-profiler.log||Profile metrics for operations performed in vCenter Server|
|vmware-vpx\vpxd-alert.log||vpxd/vpxd-alert.log||Non-fatal information logged about the vpxd process|
|perfcharts\stats.log||perfcharts/stats.log||VMware Performance Charts|
|eam\eam.log||eam/eam.log||VMware ESX Agent Manager|
|invsvc||invsvc||VMware Inventory Service|
|netdump||netdumper||VMware vSphere ESXi Dump Collector|
|vapi||vapi||VMware vAPI Endpoint|
|vmdird||vmdird||VMware Directory Service daemon|
|vmsyslogcollector||syslog||vSphere Syslog Collector|
|vmware-sps\sps.log||vmware-sps/sps.log||VMware vSphere Profile-Driven Storage Service|
|vpostgres||vpostgres||vFabric Postgres database service|
|vsphere-client||vsphere-client||VMware vSphere Web Client|
|vws||vws||VMware System and Hardware Health Manager|
|workflow||workflow||VMware vCenter Workflow Manager|
|SSO||SSO||VMware Single Sign-On|
It would be simpler again to use a program like Log Insight to help you parse through the logs. And you wouldn’t need to download them as they are being streamed to Log Insight. You’ll see output similar to what I mentioned above.
Objective 4.2 – Create and configure vSphere objects
Creating and configuring objects can be done several ways. You can do this through the HTML5 client, or you can do this from the CLI using PowerCLI or use commands at the ESXi SSH prompt. Inside the HTML5 client it is as simple as right clicking on the parent object (such as a cluster) and then selecting Add Host or New Virtual Machine. This is the window you may see when you right click on the parent object:
Configuring an object depends on the object. Configuring a VM is as simple as right clicking on it and Configuring Settings. You can also select the object and then use the center pane to bring up the Configure pane. This may give you different options to configure based on the object. Here is a screenshot of the Configure pane for a ESXi host.
As you can see there are a number of ways to accomplish this task.
Objective. 4.3 – Set up a content library
Setting up a content library is straightforward. To do this:
- Click on Menu at the top of your screen and then select Content Libraries
- Click on the ‘+’ to add a new Content Library
- Specify a Name for the library and any notes. Also if needed change what vCenter Server you will host this off of.
- This screen has options for how you want to use it. This can be setup as a Local or you can Subscribe to someone else’s library. If you do create a local library, do you want others to be able to subscribe to it. If publishing, will they need to authenticate.
- You need to store the Content Library somewhere. You do that on this screen.
- That’s it! Click Finish
Objective 4.4 – Set up ESXi hosts
Pre-requisites was gone over in Section 1, so I imagine if you got to this point you already know those. You can install ESXi several different ways.
- Interactive Installation – this is you sitting at a console or in front of the server and running the installation. This can be installed from an ISO file, USB stick, CD-ROM, or PXE. The actual installation is fast and straightforward, taking about 15 min or so.
- Scripted Installation – This is more efficient than the interactive as you can do many more at the same time and you aren’t required to answer prompts. The prompts are filled out automatically by an unattended file. The installation script needs to be stored in a location that the host can access with HTTP, HTTPS, FTP, NFS, CD-ROM, or USB.
- Auto Deploy Installation – This can provision hundreds of machines at the same time. This can be setup to use a remote disk and can store that setup locally or pull it down every time the machine boots. These options are known as Stateless Caching and stateful installations. With Auto Deploy you create a host profile that allows you to configure the host with specific things like Virtual Standard Switches with a specific name etc. This is great for enterprise because it allows you to keep a standard image and settings.
Once the machine is setup you can further configure it using the configure pane as we saw in Objective 4.2 (screenshot). This allows you to change options such as NTP and more. These settings could be setup if using host profiles.
- To add hosts in vCenter Server, you first must have a Datacenter. You create that by right clicking on the vCenter Server and choose New Datacenter
- After that is created, you can right click on the Datacenter and Add Host.
- Enter the IP or Fully Qualified Domain Name (FQDN). Make sure it can be resolved by DNS
- Enter connection details for username and password
- You are asked to check the certificate and after approving it, you will be given a summary
- Assign a license to it
Assign a lockdown mode if you want to use it
- Assign where you want to put the VMs from this host (if there are any on it)
- Click Finish and Complete it.
Objective 4.5 – Configure virtual networking
You configure virtual networking different ways, depending on your environment. Configuring VSSs can be done using the ESXi HTML5 client as seen here
Physical NICs are how you access your Physical Network. You create VMKernel ports which are how ESXi accesses the internal switch for management tasks and you have Virtual switches to connect both together. Finally, you have port groups which is a grouping of vNICs or the virtual machine NICs. A better way to show this is with a picture.
- These are the VMKernel ports – These are used for management tasks such as vMotion etc.
- pNICS or Physical Network cards are on the other side and how you reach the physical network.
- VM Network is the name of my Port Group which is how I group all the NICs from the VMs underneath. I group them to easier perform tasks on all of them.
- The construct in the middle is my Virtual Switch. This one is a VSS
The picture above can be accessed on the host page under the configure tab. You can also make changes there. A VDS is accessed under the sub category networking by using the menu up top or corresponding icon.
The picture for VDS looks much like the one for VSS but will mention all the different uplinks on each host.
You can make changes there as well. Or by right-clicking on the actual switch on the navigation pane on the left.
Objective 4.6 – Deploy and configure VMware vCenter Server Appliance (VCSA)
This objective is the installation and configuration of vCenter Server Appliance. The installation may vary a tad depending on the type of installation you do. Here is a workflow. I am going to assume you already have at least one ESXi host setup since we covered that a couple of objectives ago. There are two workflows. One for large environments and one for smaller.
The vCenter Server UI install, whether for a vCenter Server or PSC, is a two-stage process. The installer contains files for both GUI and CLI deployments so you only need the one ISO. The first stage is deployment of the OVA file into your environment. The second stage configures and starts all the services of your shiny new appliance. The CLI is slightly different. You run a CLI command against a JSON file you have inputted your configuration parameters in. This in turn creates an OVF Tool command that deploys and configures the appliance in one go.
Once setup, you log into the appliance with the username “root” and whatever password you set while deploying. Single Sign On comes later. Lets see what the install looks like.
- For a Microsoft Windows admin station, you will mount the ISO and go to <CD-ROM Drive Letter>\vcsa-ui-installer\win32\installer.exe and double-click.
- You are then presented with this screen
- We are going to Install so click on that box. The first stage then begins.
- Click Next and Accept the End User Agreement. The next screen is where we decide what type of installation we want to perform.
- I am going to choose embedded. Notice the External PSC model will soon not be supported.
- We now need to choose the ESXi host to install to (or vCenter Server). Generally the port will be 443 unless you have changed your environment.
- Accept the Certificate warning
- Enter in the name you want to give your vCenter Server that will appear in the VM inventory. Type in a password that you want to use for the vCenter Server.
- Decide on Deployment Size and Storage Size. Keep in mind if this vCenter will be doing heavy processing you may want to upsize it. This will give it more vCPUs and memory to use.
- Select the datastore you want to install to and if you want to use Thin Disk Mode or Thick. You can also create a vSAN datastore to install to.
- Network settings now need to be entered in.
- It is now ready to complete stage 1. Let it finish.
- Stage 2 begins. You need to decide how to synchronize time and if SSH access will be open.
- You then need to create a SSO domain or join an existing one. If you create one, make sure it is not the same name as your Windows Domain as that can cause all sorts of issues. This is also where to set the password for Administrator@SSODomainyoumakeup.something.
- Decide if you and your company want to share anonymized data with VMware.
- Finish and watch it work.
That’s all there is to the setup. You can configure it when its done through the appliance setup page. This is the normal address for the vCenter Server but put :5480 at the end. For example https://vCenter.vsphere.local:5480
That page will allow you make changes to many of the parameters as you can see here.
There are quite a few setting you can set through the HTML5 UI as well as seen here.
Objective 4.7 – Set up identity sources
You can setup additional identity sources in your VMware environment to allow more granular control of permissions and for better management. You can set them up by going to the Menu at the top and clicking on Administration. Then going to configuration and adding the identity source.
An Active Directory, AD over LDAP, or OpenLDAP identity source can be used. You can use a machine account in Active Directory or a Service Principle Name to authenticate.
Objective 4.8 – Configure an SSO domain
The only real way of configuring SSO that I can find is just users. This is done from within the same place as our identity sources. Instead of configuration menu item, you click on Users and Groups right above that. This allows you to see the Users for your SSO. You then click on the 3 dots in front of the user to change/edit/delete them.
That’s it. Moving on!
Section 2 – VMware Products and Solutions
Objective 2.1 – Describe vSphere integration with other VMware products
VMware has just a few products on the market (/sarcasm), and they show no letup in acquiring other companies and expanding to new technologies. One thing I appreciate about them is their ability to take what they buy, make it uniquely theirs, and integrate it with their current solutions. While this is not always done quickly and it make take a few versions, it usually pays dividends. Other products such as their Software Defined Networking product, NSX-V and T, and vSAN (SDS storage) and more, round out their offerings making it a complete solution for their customers. While definitely not altruistic, having a single place to get a complete solution can make life easier. Let’s look at some of the VMware products that are commonly used with vSphere core products.
If you look at products grouped together on VMware’s download site, you’ll see the core vSphere products of ESXi and vCenter. You also see Log Insight, NSX, Operations, and Orchestrator. I will try to give you a high-level of each of those products and how they fit into the vSphere world.
vRealize Log Insight
vRealize Log Insight is a syslog server on steroids. It is described as a Log Management and Analytics Tool by VMware. It integrates with vCenter Server and vRealize Operations. Log Insight can be used as a regular syslog server for other solutions not in VMware. Using it as a single logging repository and being able to search across your entire company’s infrastructure is its true superpower. But wait… there’s more.
You can also load content packs to manage specific solutions. One example of this is I am using a specially created Rubrik content pack that allows me to create specific dashboards to monitor my backups. Log Insight has the ability to have multiple users and assign them separate permissions to create their own dashboards and metrics. You can see my walkthrough on Log Insight (albeit 4.3 instead of 4.6) here. I also have a few videos to show you how you might customize dashboards here and how you can track a error in the logs here.
What VMware did for Server hardware they did with Networking as well. While ESXi and vCenter Server already have VSS and VDS, this is the next step in networking evolution. Using NSX you can implement normally difficult configurations such as micro-segmentation in your datacenter with ease. Being able to do this all from a single UI makes it easy and saves time. Once the initial configuration of the physical networking is done, everything thereafter can be accomplished in VMware’s HTML5 client. Creating switches, routers, load balancers, firewalling, you name it.
Because NSX’s technology, ESXi essentially believes it is on a large L2 network allowing you to do things impossible before, such as vMotion over large geographic distances. NSX brings a lot to the table. There is a lot to learn about it, however and it has its own certification track.
vRealize Operations is a tool used to facilitate performance optimization, capacity management, forecasting, remediation, and compliance. It integrates right into the HTML5 client and keeps you constantly aware of how your environment is performing. Not only does vRealize Operations integrate with ESXi and vCenter, it also integrates with NSX and Log Insight. Here is a pic of what it looks like in the HTML5 client
I also have a few videos on how to perform actions in vSphere Operations here. While this is an old version it serves well to show you some of the things you can use vRealize Operations for.
You have a large number of dashboards to choose from and monitor. You can see things like disk usage and capacity graphically making it easy to pick out potential problems at a quick glance. Doing this paper vRealize notified I’ve been running my Plex Server on a snapshot for a long period of time… I didn’t have any idea until it told me. (Snapshot was created by Update Manager upgrade). Short story, you need this in your life.
Most people know about the app IFTTT for your phone. This is kind of like that but way more powerful. Using vRealize Orchestrator you can create workflows that can perform a plethora of different tasks. It also integrates with vRealize Automation to create even more complex jobs. Using vRealize Orchestrator, you can:
- Configure software or virtual hardware
- Update databases
- Generate work order tickets
- Initiate system backups
And much more. This integrates with all of VMware’s other products and is a drag and drop worklflow solution.
Objective 2.2 – Describe HA solutions for vSphere
We already went over this, but we’ll touch on it again. The main High Availability solutions VMware provides are vMotion, svMotion and HA using clusters. I will include both HA parts so that you can read about HA in one fell swoop.
HA works by pooling hosts and VMs into a single resource group. Hosts are monitored and in the event of a failure, VMs are re-started on another host. When you create a HA cluster, an election is held and one of the hosts is elected master. All others are slaves. The master host has the job of keeping track of all the VMs that are protected and communication with the vCenter Server. It also needs to determine when a host fails and distinguish that from when a host no longer has network access. HA has other important jobs. One is determining priority and order that VMs will be restarted when an event occurs. HA also has VM and Application Monitoring. Using this prompts HA to restart a VM if it doesn’t detect a heartbeat received from VM Tools. Application Monitoring will do the same with heartbeats from an application. VM Component Monitoring or VMCP allows vSphere to detect datastore accessibility and restart the VM if a datastore is unavailable. One last thing to note. In the past, VMware tried to trick people by using the old name for HA which was FDM or Fault Domain Manager
There are a several configuration options to configure. Most defaults work without drama and don’t need to be changed unless you have a specific use case. They are:
- Proactive HA – This feature receives messages from a provider like Dell’s Open Manage Integration plugin. Based on those messages HA will migrate VMs to a different host due to possible impending doom of the original host. It makes recommendations in Manual mode or automatically moves them in Automatic mode. After VMs are off the host, you can choose how to remediate the sick host. You can place it in maintenance mode, which prevents running any future workloads on it. Or you could put it in Quarantine mode which allows it to run some workloads if performance is low. Or a mix of those with…. Mixed Mode.
- Failure Conditions and responses – This is a list of possible host failure scenarios and how you want vSphere to respond to them. This is better and gives you way more control then in the past.
- Admission Control – What good is a feature to restart VMs if you don’t have enough resources to do so? Not very. Admission Control is the gatekeeper that makes sure you have enough resources to restart your VMs in the case of host failure. You can ensure this a couple of ways. Dedicated failover hosts, cluster resource percentage, slot policy, or you can disable it (not good unless you have a specific reason). Dedicated hosts are dedicated hot spares. They do no work or run VMs unless there is a host failure. This is the most expensive (other than a failure itself). Slot policy takes the largest VM’s CPU and the largest VM’s memory (can be two different VMs) and makes that into a “slot” then it determines how many slots your cluster can satisfy. Then it looks at how many hosts can fail and still keep all VMs powered on based off that base slot size. Cluster Resources Percentage looks at total resources needed and total available and tries to keep enough resources to permit you to lose the number of hosts you specify (subtracting amount of resources of those hosts). You can also override this and set aside a specific percentage. For any of these policies, if the cluster can’t satisfy resources for more than existing VMs in the case of a failure, it prevents new VMs from turning on.
- Heartbeat Datastores – Used to monitor hosts and VMs when the HA network as failed. It determines if the host is still running or if a VM is still running by looking for lock files. This automatically uses at least 2 datastores that all the hosts are connected to. You can specify more or specific datastores to use.
- Advanced Options – You can use this to set advanced options for the HA Cluster. One might be setting a second gateway to determine host isolation. To use this you will need to set two options. 1) das.usedefaultisolationaddress and 2) das.isolationaddress[…] The first specifies not to use the default gateway and the second sets additional addresses.
There are a few other solutions that touch more on Fault Tolerance and Disaster Recovery.
Fault Tolerance or FT creates a second live shadow copy of a VM. In the even the primary goes down, the secondary kicks in and it then creates a new shadow VM.
Disaster Recovery options include vSphere Replication and Site Recovery Manager. Both of these can be used in conjunction to replicate a site or individual VMs to another site in case of failure or disaster.
Objective 2.3 – Describe the options for securing a vSphere environment
There are a number of options available to secure your vSphere environment. We will start with ESXi and move on to a few others.
- Limit access to ESXi – this goes for both the physical box but also any other way of accessing it. SSH, DCUI, or remote console via IPMI or iDRAC/iLO etc. You can also take advantage of lockdown modes to limit access to just vCenter.
- Use named users and least privilege – If everyone is root than no one is special. Only give users that need it, access. Even then only give them the access and rights they need to do their job. Make sure they all log in as the user you give them. This allows for tracking and accounting.
- Minimize open ports – your ESXi host has a stateless firewall but if all the ports are open, it’s not providing any protection for you.
- Smart Card authentication – ESXi now supports smart cards for logging on instead of user name and passwords.
- Account lockouts – After a number of incorrect tries to log in, have the account lock.
- Manage ESXi certificates – While there is a Certificate Authority in vCenter, you might want look into using third-party or enterprise CA certificates.
- VIB Integrity – try to use and only allow your ESXi hosts to accept VMware accepted or VMware Certified VIBs.
vCenter Server Security
- Harden all vCenter host machines – make sure all security patches and the host machines are up to date.
- Assign roles to users or groups – This allows you to better keep track of what users are allowed to do if they are part of a role.
- Setup NTP – time stamps will be accurate and allow you to better track what is going on in your environment.
- Configure Single Sign On – Keep track of the identity sources you allow to authenticate to your vSphere environment.
- vCenter Certificates – remove expired or revoked certificates and failed installations.
- Protect the guest operating system – Keep your OS up to date with patches and any anti-malware or anti-spyware. Most OSs also have a firewall built-in. Use that to keep only necessary ports open.
- Disable unnecessary functionality – Turn off and disable any services not needed. Turn off things like HGFS (host-guest filesystem) that allows you to copy and paste between the VM and remote console.
- Use templates and scripted installations – After you spend all the time making an OS secure, use that as a template so that you don’t have to perform the same on the next machine. This also makes sure you don’t forget settings or configurations that may end up being disastrous. Script management of machines and installations for the same reason.
- Minimize use of the virtual machine console – Just like you would secure access to the physical machine, you should secure access and use sparingly the console.
- Use UEFI secure boot when possible – If the OS supports it, you can use this to prevent changes to the VM.
- Isolate network traffic – Separation of network traffic into segments allows you to isolate important networks. A prime example of this is creating a management network that is separate from regular VM traffic. You can perform this easily using VMware NSX or even as simple as creating a separate subnet and locking that down virtually or physically to ports.
- Use firewalls – Again using NSX this becomes really simple to create firewall and micro-segmentation. Mentioned above, you can also utilize firewalls in the OS but that can get unwieldy with 1,000s of VMs. Physical firewalls are a staple as well.
- Consider Network Policies – Switches in your virtual environment have security policies you can implement to prevent malicious attacks. These are promiscuous mode, MAC address changes, and forged transmits.
- Secure VM networking – same as above with securing OSs and firewalling.
- VLANs – These can be used to segment your network and provide additional security. This also breaks up your broadcast domain which can cut down on unwanted broadcast traffic.
- Secure connection to your Storage – Usually companies setup separate networks for their storage. This is for security but also performance. You can also implement authentication on your storage array such as CHAP. Fibre Channel is particularly secure as it is difficult to tap a fibre cable.