Its just another layer

Ansible tagging - why?

Thu, 28 Dec 2017 00:00:00 +0000

In recent times I have been utilizing Ansible nearly constantly for a variety of automated deployments from Openstack, AWS and Azure. Through this there have been a variety of different approaches I’ve run into on how to structure Ansible Playbooks. Some have unique playbooks for each different status change by having a playbook to start the enviornment and one to stop it. Others have a single extra_var that is passed in to determine the state of the environment and that handles starting and stopping the space. I’ve even seen the state of the enviornment be split out into different playbooks in different repos.

The one that I simply don’t get is using tagging in ansible for starting/stopping/installing the environment. In this design there is a single playbook that is used for all the different states an environment can be in. Example of it would be something like this from a command line:

ansible-playbook -i inventory MyApp.yml --tags start,common,assigntags,security

In this you run everything that is marked with start, common, assigntags and security. Along with that the un-tagged items always run.

Pros to using Tagging:

Allow skipping over a given sections (plays or tasks) of the playbook

Cons to using Tagging:

Easy to create spagetti code
Difficult to debug as more tags are introduced and their interactions

Example:

[...]
- name: My first action
  hosts: appserver
  roles: app-make-it-go
  tags: 
      - common, security, start
      
- name: My second action
  hosts: appserver
  roles: app-configure
  tags:
      - common

Now this says if you pass in the security tag, only the first play will execute.

What if the role has some tasks that are marked with common and you didn’t pass that in to that first task?

How does the caller know that some steps won’t run while others will?

I spent 6 hours debugging one odd issue I was having working with an existing playbook that utilized tags and ultimately I ended up removing all the tags and creating purpose built playbooks to run instead. This allowed me to follow the logic clearly and find that the issue was one caused because the role had tags and was expecting a certain state when that task ran. Because of the tags used, that state was skipped in a variety of situations.

I highly encourage folks who build ansible playbooks to think two, three, four, five times about using tags. It might make it easy when the playbook is first created when everything and every flow is in your head. It will more likely make whoever has to fix have to unwind the complexity added to the playbook to save a couple files.

Please.. think of the maintainers.

fsck hfs+ filesystem on ubuntu

Fri, 25 Sep 2015 02:04:35 +0000

My wife's laptop backup hard drive is starting to die. Now that hard drive is only about 5 years old, firewire external drive. So I think we've definitely gotten our use out of it. Needless to say she's a little concerned about loosing some of the data on it, so I told her to copy it over to the home file server that gets automatically backed up to the cloud. She proceeded to start copying the data over and since this hard drive is having a rough time, the GUI copy kept failing as some of the files are corrupted now.

So I took the hard drive down and directly connected it to my Ubuntu system. My drive came up as /dev/sdg. First step is to get hfsprogs installed

apt-get install hpfsprogrs

Once it is installed, to be safe I wanted to run fsck on the filesystem to give me the best chance possible to copy the data over. Next I ran

fdisk -l

[...]

Disk /dev/sdg doesn't contain a valid partition table

Now that doesn't surprise me since mac uses GPT by default and this drive is a little wonky. Instead I used GNU Parted

parted

(parted) select /dev/sdg

Using /dev/sdg

(parted) print

Model: Ext Hard Disk (scsi)

Disk /dev/sdg: 500GB

Sector size (logical/physical): 512B/512B

Partition Table: mac

Number Start End Size File system Name Flags

 1 512B 32.8kB 32.3kB Apple

 2 32.8kB 61.4kB 28.7kB Macintosh

 3 61.4kB 90.1kB 28.7kB Macintosh

 4 90.1kB 119kB 28.7kB Macintosh

 5 119kB 147kB 28.7kB Macintosh

 6 147kB 410kB 262kB Macintosh

 7 410kB 672kB 262kB Macintosh

 8 672kB 934kB 262kB Patch Partition

10 135MB 500GB 500GB hfs+ Untitled

Now I know that I'm looking to run fsck on partition 10.

fsck.hfsplus -f /dev/sdg10

Once that completed successfully, I mounted it with

mount -t hfsplus /dev/sdg /media/usbdrive

Finally a way to allow this to run a couple times and not just stop on the first failure due to Read Errors or IO errors I used

rsync -avE /media/usbdrive /media/backupsystem

Enjoy.. HFS+ support is very good in linux from my experience now. This copy took about 12 hours with many retries to get what I could off of it.

Windows 10 GA VMware Template

Thu, 17 Sep 2015 02:02:51 +0000

An interesting change with App Store and Windows Update has made creating a VMware template a little more unique now. I attempted to create one doing the normal process of building from ISO, logging in, making the modifications we want to have done such as installing VMware Tools and then gracefully powering the system off. Then the Deploy from Template option and select a guest customization to Sysprep the system. That doesn't work as any Windows Store application such as the start menu or the Windows Store, simply doesn't start.

VMware says Windows 10 guest customization works just fine per the Guest OS instruction page.

http://partnerweb.vmware.com/GOSIG/Windows_10.html

This is a tad misleading as this set of instructions does not allow you to actually get SysPrep with a Guest Customization option to run correctly. To make SysPrep work with Windows 10, the Windows 10 system needs to be put into Audit mode.

The easiest way to enter audit mode when setting up a VMware template (at least a way it worked for my setup).

Provision a blank system from DVD.
When the system boots for the first time at the Welcome Screen/Customize Privacy settings, press + + .

Make the legacy configuration settings and setups (such as creating a local ID you want on the system using Computer Management or other fun legacy non-Windows Store apps).

Do a graceful shutdown.
Turn the system into a template

This worked very well for my setup. Any better ways to setup a clean Win10 template?

Windows 10 Build 9926 Sysprep Fails

Mon, 16 Mar 2015 21:44:35 +0000

I've been working on getting Windows 10 available in our private cloud as blueprint and kept having the deploy fail when trying to sysprep it. When logging in directly to the system and running sysprep, we see this error.

When digging further we find in the C:\Windows\System32\Sysprep\Panther\setuperr.log this text.

SYSPRP Package Microsoft.Getstarted_1.5.22.1_x64_8wekyb3d8bbwe was installed for a user, but not provisioned SYSPRP Failed to remove apps for the current user

This is caused by Windows App Store installing things while the template is online and connected to the network.

Resolution

Two solutions I've found so far

When installing the OS, have it disconnected from the network.
Clean up the template if that doesn't work by running the commands
```
get-appxpackage -name *packagename* | remove-appxpackage
```

Monster VMs Mass Storage & How to check storage used

Wed, 07 Nov 2012 00:00:00 +0000

Jason Boche posted an ESXi Heap Size interesting discovery with regards to Monster VMs and the amount of storage one would be sticking to them. One of the large projects being designed today has the likely reason to hit this limit with just a single VM. This is pretty easy for that project to hit that as a single VM will have about 75TB assigned to it. Since this design has 4 or 5 VMs of this size, which then means our standard ESXi Host will have around 300+ TB assigned to it. Thankfully RDM isn’t impacted by this limit in theory, so there is a workaround available. Not ideal and yet doable.

Instead one of the questions that came out of this news is “Does this issue impact the rest of our environment?” Will this explain some random stability issues we have seen?

This script will return what every single host has in terms of VMDK files. To use this script look at each host and see if the host itself is under the limits. Then look at the cluster level based on the HA rules and see what the VMDK limits would be if a host failed. That should give an idea of how close your clusters might be to hitting this limit.

if ( -not (Get-PSSnapIn | where {$_.Name -eq "VMware.VimAutomation.Core"}) )
{
    Add-PSSnapin -Name "VMware.VimAutomation.Core"
}
$vcenters = "vCenter"
connect-viserver -server $vcenters -cred (Get-Credential)
$diskuse = @()
$clusters = get-cluster
foreach ($cluster in $clusters) {
    "Working on Cluster $cluster"
    $vmhosts = get-vmhost -Location $cluster
    foreach ($vmhost in $vmhosts) {
    "`tWorking on VMHost $vmhost"
        $vmlist = get-vm -location $vmhost
        $totalDisk = 0
        foreach ($vm in $vmlist) {
            $view = $vm | get-View
            foreach ($disk in $view.Guest.Disk) {
                $disksize = ([math]::Round($disk.Capacity/1MB))
                $totaldisk += $disksize
            }
         }
         "`t`tVM size: $totaldisk"
            $obj = new-object PSObject -Property @{
                Cluster = $cluster
                VMHost  = $vmhost
                TotalDiskMB = $totaldisk
            }
         $diskuse += $obj
    }
}
$diskuse | export-csv "OpenFileDiskUse.csv" -NoTypeInformation

With Default settings the magic numbers to look for:

ESXi 4.x - 4TB
ESXi 5.1 - 8TB

EUC1453 - Managing from the Middle with Horizon

Thu, 30 Aug 2012 20:08:29 +0000

The Post-PC world is not Mac vs Windows. It is all about the changes of multi-device, data mobility & anywhere access. As such applications and means of access need to change.

Complexity for end clients has gotten much more complicated due to multiple devices, types of applications and so on. So the management of the environment is very complex behind the scenes. The requirements for cost management, easy to use & security still exist. Those basic back end necessities are crucial to IT.

How did this happen? Its the piecemeal approach. IT had to get a tool to manage it for the Mac. Then had to manage it for iOS. Then Android and so forth. This is for Web tools, SaaS tools, Mobile Apps etc.

Consumeration of IT really means it has to be easy or the end user will go somewhere else. Now to deliver this, VMware's approach is to manage from the middle. My Apps, My Files, Native Experience.

Now Horizion suite is aiming to managed in the middle and be the portal to facilitate access and control for all the various end user computing space to the application and devices. By doing this a Catalog can be offered, file sharing can be offered and controlled. Central management is required to offer security policy controls also.

The three ways of managing in Horizon view point is Identity, Policy & Context. In Identity, who are you. Context is what are you trying to access and from and what OS. This is different than before where the PC was a known entity. Now there are more choices and options here.Policy takes Identity and Context to apply rules across both of them.

Horizon offers a place to manage, security, track, delivery and make functional. Their goal is to encourage them to opt-in instead of force down. End customers will find a way around IT wherever possible. Getting them to buy in is the future.

INF-SEC1840 - ESXi Hardening Guide and Security Practices

Thu, 30 Aug 2012 18:09:03 +0000

The ESXi Hardening guide started in the 2.x days. It was originally a best practices document around security. In the 4.x, VMware started breaking it into categories of information. Improvements over the years. Some feedback on the 4.x guide is that it was presented in PDF, limited ways on how to resolve it and mitigate the listed risk and was not usable in a programmatic format.

The new format in 5.0 is a spreadsheet guide with categorization by component and sub-components. Now this spreadsheet includes techniques to implement and apply these recommendations. Look at http://vmware.com/go/securityguides for more information.

Automating these solutions and suggestions is the next obvious step in addressing the security concerns. As such the SCAP standard from NIST which allows security audit and validation. This Standard allows you to do checklist validation style approaches and how it is setup. XCCDF is the primary XML rule based system for validation around testing and assessment. XCCDF is the setup of the checklist approach and then OVAL designates a fixtext area which can be programmatic or just manual steps. Together these two things will help by having standardized approach to building and utilization of security information and validation.

OVAL is the Open Vulnerability Assessment Language. Today there are over 13,500 definitions made that include different versions of all OS platforms. This is open and community driven and significant amount of information being created every day. This standard links with CVE to show vulnerability scoring which is more timely and updated reasonably quickly.

INF-VSP1365 - Software Defined Security & Networking

Tue, 28 Aug 2012 18:58:42 +0000

The 4 main layers of making Software Defined Networking & Security work from Bottom Up is

Abstraction
Pooling
Service Insertion
Administration

Today our networking and security structures are built around the physical entities that we setup. As such the design is tied to the physical entities and how they are setup. In the HyperVisor world, where VMs and services can be tied together into virtual datacenters, the physical entities are the limiting factors to better containment of security.

With Software Defined Networking, it is now possible to take that set of VMs which are tied together into a vApp container and apply security/network together around that container. Not just tied to the physical or IPs, can be the entirety of the VM container. This changes the conversation of security to talking about the smallest realistic entity of the OS instance instead of just one property of that OS instance (being IP address).

VMware has been working with some integrations in this space with F5, RiverBed, Symantec, SafeNet, Brocade, Emulex as a short list.

f5, one of the leaders of load balancing, is helping speed up application provisioning. Today it can easily take 4+ weeks in some businesses due to the internal process of throwing over the wall with high failure rates. Setting up the policy for infrastructure and then selection/application of the policy from the service catalog. f5 has reduced down to 25 mins to fully automatically deploy these rules. f5 does this by integration of their Enterprise Manager through the vShield Manager API.

RiverBed Cascade product is another integration point with vXLAN. One of the big challenges is the Loss of Network Visibility & Control. RiverBed can do this with IPFIX monitoring in Q4 this year. They can dig into the vXLAN protocol with deep inspection. This does it by reading into the Virtual Distributed Switch. It gives Performance Management for the Software Defined Networking. It can break down and work across the multi-tenancy design.

Symantec who is a heavy weight for security has some heavy integrations and some overlays. Not much to be said here and depends on which product line that is examined.

The thing that makes this possible is the simplicity of the integration capabilities. For many of these Partners they can continue to focus heavily on their products and be able to create a view into their data via a vCenter Plugin reasonably easily.

Intuit, maker of Quicken, has been working hard to delivery of Software Defined Datacenter. They have many steps from R&D to production. They go from Dev -> Test -> Performance -> Pre-Production -> Production -> Ongoing Compliance. As such they need to support this on top of adding security, stability and HA/DR needs. And of course lower the costs. At the end the goal is to provide IT Agility.

Legacy designs meant they had to create unique zones of systems include physical routers and firewalls to handle public, private, dev and sensitive compartmentalization. Very slow and difficult to provision new systems for teams to use. The average app is a 3-Tier application. Very high CapEx costs and there is no isolation within each zone. This set of over the wall throwing of tickets means it takes 3+ weeks at best to get something setup.

The new design allows Intuit to pool everything together into a virtual hosting zone for the business to get systems from. This software defined datacenter is their new solution. They capture all the inforamation in a "blueprint form". They include VM, Storage, ACLs, Network and various customizations needed. This gets fed into the request and is built out for them. From there they have distinct provider zones that are compartmentalized. By doing this set of abstractions they have 3x CapEx improvement, 2x Density, secure multi-tenancy and it is self service based. This now takes on average about 30 mins to deploy from start to finish.

This change in viewpoint and approach has allowed IT at Intuit to turn from IT Provider to Customer Enabler. This has given significant change to what IT folks have been doing in their day to day jobs.

Keynote Day 2 - Herrod and Future of End User Computing

Tue, 28 Aug 2012 16:51:05 +0000

Today's Keynote is all about the End User Computing experience and the Battle of the Platinum partners.

Branch in a box through the View Rapid Deployment Program. Take an appliance, some configuration and deploy as many workstations as you need.

Mirage from Wanova is being presented as the future of secured solutions and centralized management. ACE, Local Mode are great solutions for a limited set of end use cases. Mirage is aimed at handling all the various other end devices out there. It will offer disaster recovery and centralized management.

After an entertaining canned demo around Mirage. They presented some more advanced demos of using Tablets and existing OS instances. Project AppShift is some R&D making it more swipe style based interface for Windows 7. This is User Interface Virtualization Techinques. This took takes several of Windows basic interface experiences and made them Tablet friendly with swiping and copy/paste across the system.

Horizon Suite Administration is announced today and a quick demo. One of the points is that Horizon can manage Xen App Applications. Horizon Mobile on iOS will wrap applications into a secured workspace. This separates applications and control around security policies. So now it allows you to manage end devices cleanly and safely across multiple devices from a single interface tool.

VMworld Challenge -What is the things that Partners are doing to improve the VM space. They get 4 minutes to give their presentation. Then everyone at VMworld will vote using the mobile app on who gave the best preso (or is doing the most interesting thing). The winner will have their charity get a sizeable donation from VMware.

What are partners doing

Cisco - Playing for Kaboom who help build playgrounds in dense city centers for kids.

Techwise TV is presented by Cisco to making Networking more close. Gist of L.I.S.P. An interestingly cute little preso around VM Mobility with all the different means to move VMs around between datacenters due to IPv4/IPv6. LISP is a free technology to use from Cisco.

Dell - playing for Girl Scouts of America.

Dell vStart 1000 is a stack that gives you the full Dell solution from storage, networking and compute in a rack with a simple management interface. Fully integrated

EMC - charity is Wounded Warriors

EMC believes that more things should be built in or easily available Directly from the Web Client, Chad Sakac from EMC, he created a backup job directly in vSphere environment. His demo is live as he does things and clicks it isn't pre-recorded. Chad ran out of time. He was the first and brave.

HP - playing for Big Brothers and Big Sisters

HP showed how they have integrated their HP Matrix Infrastructure orchestration suite and how it works with vCloud Director. It organizes and automates the integration and backend creation of a provider DC.

NetApp- Playing for Be The Match, helps DNA matching for bone marrow

Data ON Tap infrastructure is the demo. How do you demo that per Dave Hitz. Peak Colo is going to be used an example of how NetApp helps them out. Everyone in Peak Colo gets a vSAN since all the infrastructure is shared overall and the vSAN is for each customer.

NetApp won and VMware is donating $10,000 to Be The Match.

Keynote VMworld 2012 - Day 1 - Right Here Right Now

Mon, 27 Aug 2012 17:16:42 +0000

In the past I did live blogging and it was an interesting thing to do. Realistically it wasn't a great experience for those watching and for me writing it. Now a days, they are streaming live with VMware NOW and with near instant Twitter responses. Instead this is more for me to document and enjoy the event.

Stomp (like) starts off the event. Loud, exciting with drums in the VMworld at the center stage. Never to let down a good show, VMworld hit it off great.

Rick Johnson, CMO start off with presentation around VMworld Agenda, General Session info and other scheduling stuff. Tuesday keynote will be covering the future of End User Computing by Scott Herrod. Then the battle royale among the Platinum Sponsors, live demo presentations with live voting of the event using mobile apps. The best live demo wins by individual votes.

VMUG Rocks the house with some Green Shirts.

Paul Maritz now takes the stage. Some interesting numbers on how the world has changed from 2008 to 2012. Basically his history on VMware. ~25k VCPs to ~125K, Workload Virtualization 25% to 60% of all loads. One of his great contributions is the Cloud direction.

For the past 30 years, the primary driver of change is moving from Paper to Computer and that was considered innovative. This generation is more about streams of data and consumption of that data in different ways. It is about real-time versus specific static reporting. Tying this data to how it impacts a person day to day is the mobile/social link.

One consistent direction for VMware coming from Paul is the 3 categories. Server to Cloud, Existing Apps to Apps & Big Data and PC to Multiple End Points. These three categories are the same breakouts of Server Layer, Development Environment & End User Computing as they have been for the past several years.

Paul at this point hands over Pat Gelsinger as the new CEO. Interesting note Pat stated is that Pat was the R&D director and had a short meeting at an Intel Conference with Mendel from VMware. They held up the conference 10 mins as Mendel was telling Pat about VMotion. Mendel had it working and they were both so excited about use cases for this.

Software Defined Decenter

Pat had said back in 2007 that VMware needed to setup the Virtual Datacenter. Significant progress has been made over the years. In order to do this the stacks and environment needs to be completely automated and virtualized.

vCloud Suite

This suite includes many of the pieces to get the Software Defined Datacenter.

vCloud Director
SRM
vSphere
vCOPs
APIs

Comprehensive, Highest Performance and Proven Reliability is the only way to put Mission Critical on something. vSphere 5.1 offers all this as the 9th major release.

vRAM is gone. Heard loud and hear and listen carefully. Going to a single CPU approach licensing model with no limits. Woot!.

Cloud OPs

So how do you keep all this stuff running and build it?

Process for Operational Readiness
Role Based Certifications
Training expansions

Multi-Cloud is the future and today.

VMware already covers the PaaS with Cloud Foundry. Now to handle Automation & Orchestration layer they have added Dynamic Ops. Finally to add the bottom is at the Software Defined Datacenter layer is Nicira for Networking Virtualization. Finally VMware joined the OpenStack organization. They have a strong spread to cover many of the layers.

New Application Environment

vFabric and Cloud Foundry is going well. Nothing really to note as new here. Just continued development and movement forward.

End User Computing

Wanova/Mirage to manage PCs both physical and virtual. Horizon is turning into the central portal for all application space access. More to come here.

Pat is very respectful over what Paul Mariz has given to VMware and the kind of visionary VMware needed. Both of these men have been revolutionary IT leaders over the past 30 years.

On to Scott Herrod's presentation.

One of the steps forward in the Server Virtualization space is moving from a single VM up to multiple VMs in a single Virtual Datacenter. To do this compute, storage, networking and management have to be included.

In the compute space in 2011 with vSphere 5, 32 CPUs and 1Million IOPS per host. Now with vSphere 5.1, 64 CPUs and 1 Million IOPS per VM. Yes that's right.. Per VM!!! Serious improvements.

Epic Medical Systems is now offering their medical critical applications on x86 and will ONLY support it on vSphere. This is an amazing announcement.

Along with all of the advancements of vSphere to help low latency and jitter sensitive applications such as Telephony.

Hadoop got some time around Project Serengeti. This is a management application for Hadoop.

Software Defined Datacenter Storage

Interesting that Scott flew over the Storage VMotion that can be done with no shared storage. This is huge as this is a major marketing capability of Hyper-V 3.0. Lots of advancements and announcements here too. Virtual Volumes, Virtualizing Flash, Virtual SAN.. SDDC for Storage is changing this world pretty seriously.

Software Defined Datacenter Networking

Many of the issues that happen now is networking. To configure and setup Load Balancing, Network configurations can be significantly complex. How do we move this along?

vXLAN ecosystem is how Scott is talking about this. There are all sorts of different networking capabilities and I just can't keep up with this all. This space is moving fast from server offload to layer 3 edge interfaces to all sorts. It might finally to be time to make the VM that goes around the world.

After having all this great tech the next challenge is how to manage it. Going forward with the Suite's will have vCOPs for example and how to integrate it in. A good example is vCops shown directly in the vCloud interface.

Ultimately being able to expand this space is the partner enhancements. So being able to utilize the existing API suite (vCloud API) to not have to deal with permissions separately or structure. Just use the existing space.

The finish up is the cool demo on the ability to setup a VDC in a couple minutes between the private and public cloud. Sounds good.. just a big techie and quick with no time to sink in.

Slipped in quickly.. Enterprise+ gets a free upgrade to the vCloud Suite.

VMware is pushing the world forward. I haven't been this wowed since VMotion was first announced.