Have you ever just wanted an IF Function when creating Super Metrics? Good news, there is one!
Leading on from the last post I did on determining the number of VMs which will fit into cluster, I have decided to further expand it with an IF function to take the Host Admission Policy failure to tolerate level into account as well.
Previously we used a flat 20% overhead as that was the company policy, however that reserved way too many resources on larger clusters, and setting it to a flat two host failures
We wanted to set any Cluster Compute Resource with less than 10 hosts, to only allow for a single host failure, but clusters of 10 and above should allow for two host failures.
In vROps terms this requires a Ternary Expression, or as most people know them, an IF Function.
You can use the ternary operator in an expression to run conditional expressions in the same way you would an IF Function.
I had a requirement from a customer to identify log events in order to create alerts for several threat scenarios. This post is intended to provide a high-level description of the results for the scenarios for future reference or in case anyone finds a use. Please see the earlier post on enabling additional vCenter and PSC logging. http://www.caenotech.co.uk/vmware/configuration-of-rsyslog-on-vcsa-and-psc/
Access to vCenter Administrator role
The objective of the following is to ensure nobody other than certain colleagues have access to the Cryptography operations within vCenter and that all work carried out on crypto operations is done under suitable change control.
As can be seen the default syslog details the Administrator user logging in as VSPHERE.LOCAL\Administrator and the IP it has originated from
<datetime> <vCenterHostname> vcenter-server: User <Domain>\<Username>@<IPAddress> logged in as JAX-WS RI 2.2.9-b130926.1035 svn-revisions#<UID>
<datetime> <vCenterHostname> vpxd <eventID> - - Event [<LineID>] [1-1] [<datetime>] [vim.event.UserLoginSessionEvent] [info] [<Domain>\<Username>] [] [LineID] [User <Domain>\<Username>@<IPAddress> logged in as JAX-WS RI 2.2.9-b130926.1035 svn-revisions#<UID>]
<datetime> <vCenterHostname> vcenter-server: User <Domain>\<Username>@<IPAddress> logged out (login time: <datetime>, number of API invocations: <x>, user agent: JAX-WS RI 2.2.9-b130926.1035 svn-revisions#<UID>)
<datetime> <vCenterHostname> vpxd <eventID> - - Event [<LineID>] [1-1] [<datetime>] [vim.event.UserLoginSessionEvent] [info] [<Domain>\<Username>] [] [LineID] [User <Domain>\<Username>@<IPAddress> logged out (login time: <datetime>, number of API invocations: <x>, user agent: JAX-WS RI 2.2.9-b130926.1035 svn-revisions#<UID>)]
the text strings “vim.event.UserLoginSessionEvent” and “vim.event.UserLogoutSessionEvent” can be used to alert on people logging into the vCenter
Alteration of vCenter Roles
Creation of a new vCenter role “newCryptoRole”
From the default log we can show that the new role is created however does not show whom by or which permissions it is given.
<datetime> <vCenterHostname> vcenter-server: New role <roleName> created
<datetime> <vCenterHostname> vpxd <eventID> - - Event [<LineID>] [1-1] [<datetime>] [vim.event.RoleAddedEvent] [info] [] [] [LineID] [New role <roleName> created]
This is where the additional vpxd-svcs log is required for details of who completed the action and what permissions were assigned to the role
[tomcat-exec-176 INFO AuthorizationService.AuditLog opId=] Action performed by principal(name=VSPHERE.LOCAL\Administrator,isGroup=false):Add role Id=-922973018,Name=newCryptoRole,Description=,Tenant=Privileges=[System.Anonymous, System.Read, System.View, Cryptographer.Clone, Cryptographer.Encrypt, Cryptographer.Migrate, Cryptographer.RegisterVM, Cryptographer.ManageKeyServers, Cryptographer.Decrypt, Cryptographer.AddDisk, Cryptographer.ManageKeys, Cryptographer.ManageEncryptionPolicy, Cryptographer.Access, Cryptographer.Recrypt, Cryptographer.RegisterHost, Cryptographer.EncryptNew]
Modification of permissions to any vCenter role
<datetime> <vCenterHostname> vcenter-server: Role modified
Previous name: <roleName>, new name <newRoleName>
Added privileges: <privilegesAdded>
Removed privileges: <privilegesRemoved>
<datetime> <vCenterHostname> vpxd <eventID> - - Event [<LineID>] [1-1] [<datetime>] [vim.event.RoleUpdatedEvent] [info] [] [] [LineID] [Role modified
Previous name: <roleName>, new name <newRoleName>
Added privileges: <privilegesAdded>
Removed privileges: <privilegesRemoved>]
From the default log we can show that the role is modified and which permissions have been added, however does not show whom by. This is where the additional vpxd-svcs log is required for details of who completed the action
[tomcat-exec-17 INFO AuthorizationService.AuditLog opId=a794037d-a725-4b89-ab96-d3a23a58648c] Action performed by principal(name=VSPHERE.LOCAL\Administrator,isGroup=false):Update role Id=-922973018,Name=newCryptoRole,Description=,Tenant=Privileges=[System.Anonymous, Cryptographer.Clone, Cryptographer.Encrypt, Cryptographer.Migrate, Cryptographer.RegisterVM, Cryptographer.ManageKeyServers, Cryptographer.Decrypt, Cryptographer.AddDisk, Cryptographer.ManageKeys, Cryptographer.ManageEncryptionPolicy, System.View, Cryptographer.Access, Cryptographer.Recrypt, Cryptographer.RegisterHost, System.Read, Cryptographer.EncryptNew, Network.Assign, Network.Config, Network.Move, Network.Delete, Task.Create, Task.Update]
From the default log we can show that the role is removed, however does not show whom by. This is where the additional vpxd-svcs log is required for details of who completed the action
[tomcat-exec-2 INFO AuthorizationService.AuditLog opId=c0100be8-9114-4e60-9520-4cf1b6015793] Action performed by principal(name=VSPHERE.LOCAL\Administrator,isGroup=false):Delete role -922973018
Assignment of User to a Role
Assigning a user to a role is not recorded in the default logs, this requires the additional vpxd-svcs log
[tomcat-exec-232 INFO AuthorizationService.AuditLog opId=] Action performed by principal(name=VSPHERE.LOCAL\Administrator,isGroup=false):Added access control [ Principal=Name=VSPHERE.LOCAL\newCryptoUser,isGroup=false,roles=[-922973018],propogating=true ] to document urn:acl:global:permissions
If you attempt to assign a user to a role with higher permissions that your current user you will receive the following error message in the vCenter Web UI
Additionally the following event is recorded in the vpxd-svcs.log
[tomcat-exec-293 WARN com.vmware.cis.authorization.impl.AclPrivilegeValidator opId=] User VSPHERE.LOCAL\newUser does not have privileges [System.Anonymous, Cryptographer.Clone, Cryptographer.Encrypt, Cryptographer.Migrate, Cryptographer.RegisterVM, Cryptographer.ManageKeyServers, Cryptographer.Decrypt, Cryptographer.AddDisk, Cryptographer.ManageKeys, Cryptographer.ManageEncryptionPolicy, System.View, Cryptographer.Access, Cryptographer.Recrypt, Cryptographer.RegisterHost, Authorization.ModifyPermissions, System.Read, Cryptographer.EncryptNew] on object urn%3Aacl%3Aglobal%3Apermissions
Adding user to Platform Services Controller SSO Groups
In order to capture logs showing adding user to the “SystemConfiguration.BashShellAdministrators” group we require the additional logs ssoAdminServer.log and vmdir-syslog.log
./sso/ssoAdminServer.log:
pool-4-thread-1 opId=73c87e6b-746c-46f2-9b59-a5da95f5a1c1 INFO com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl] [User {Name: Administrator, Domain: vsphere.local} with role 'Administrator'] Adding users to local group 'SystemConfiguration.BashShellAdministrators'
./vmdird/vmdird-syslog.log:
info vmdird t@139993972463360: MOD 1,add,member: (CN=Administrator,CN=Users,DC=vsphere,DC=local) info vmdird t@139993972463360: Modify Entry (CN=SystemConfiguration.BashShellAdministrators,DC=vsphere,DC=local)(from 127.0.0.1)(by <PSCName>@vsphere.local)(via Ext)(USN 4974)
Cryptographic Components
The objective of these alerts are to ensure that vSAN encryption is not disabled (where enabled) or enabled (where it’s not). Equally, any tampering with KMS (required for encryption) should be correlated back to change control / incident management.
As user with “Administrator – No Cryptography” if you try to disable encryption on vSAN they do not receive the option due to a lack of privileges
Disable vSAN Encryption
In this test, vSAN encryption was disabled. This is considered a reconfiguration of vSAN and logged accordingly.
Default vCenter logs show that vSAN is
being reconfigured:
ESXi Host logs show that on the string [VsanSystemImpl::Update] the vSAN is being reconfigured and has encryption set to ‘enabled=true’.
Adding a KMS Server
The event of adding an additional KMS is logged, specifying the KMS alias name and the KMS Cluster into which it is added.
<datetime> <vCenterHostname> vpxd <eventID> - - <date> info vpxd[<Rand>] [Originator@xxxx sub=CryptoManager opID-KmipServerPageMediator-add-xxxxx-ngc:<rand>] A new Kmip Server <KMSName> is registered in cluster <KMSCluster>
The string “A new Kmip Server” can be used to alert on a new KMS server being added to the KMS Cluster.
Delete a KMS Server
The KMS Server was unregistered from the
VMware vCenter.
The following event described the removal.
<datetime> <vCenterHostname> vpxd <eventID> - - <date> info vpxd[<Rand>] [Originator@xxxx sub=CryptoManager opID-KmipServerActionResolver-remove-xxxxx-ngc:<rand>] Kmip Server <KMSName> is removed from cluster <KMSCluster>
vMotion
vMotion a VM from vSAN Datastore to Local Storage
The Test Virtual Machine (permbound1) was migrated from vSAN ‘vSANDatastore’ to local storage named ‘ds-local-ESXiHostnameLocalDS’
The following events were recorded by the default vCenter logs.
vcenter-server: Migrating <VMname> from <ESXiHostname>, <datastoreName> to <ESXiHostname>, <datastoreName> in <vCenterDatacenter>
The event is in the format and notes the time, who carried out the migration under the field “vc_username”, what was migrated, and the source/destination hosts and datastores.
As part of a client’s environment, there was a requirement from the end customer to forward additional logging information above the default logs forwarded by vCenter Server and Platform Services Controller (PSC).
In order to provide these additional logs
configuration of rSyslog is required to specify these files.
This post is intended to provide steps to implement these changes.
Additional logging available from non default vCenter logs
Single Sign-On Activities
Successful SSO Login
Successful SSO Logout
Successful SSO Active Directory Login
Successful SSO Active Directory Logout
Failed SSO Login
Failed SSO Login (User not found)
Failed SSO Active Directory Login
Failed SSO Active Directory Login (User not found)
SSO User Creation
SSO User Password Change
SSO User Deletion
SSO Group Creation
SSO Group Assignment
SSO Group Deletion
SSO Password policy update
vCenter Server Activities
Successful vCenter Server Login
Successful vCenter Server Logout
vSphere Permission Created
vSphere Permission Updated
vSphere Permission Deleted
vSphere Role Creation
vSphere Role Update
vSphere Role Deletion
In order to capture the above activities, you will need to forward the following log files:
/var/log/vmware/sso/vmware-sts-idmd.log
/var/log/vmware/sso/ssoAdminServer.log
/var/log/vmware/vpxd-svcs/vpxd-svcs.log
/var/log/vmware/vpx/vpxd.log
NOTE: I am not including the vpxd.log in my implementation below as it is an extremely verbose log and we did not require it for the security events we wished to capture. Additionally I don’t want someone blindly copying the config below without understanding it and accidentally upsetting their environment.
Implementation Steps
VMware Appliance Management Interface (VAMI)
Step 1 – Connect to the VAMI interface
for all vCenters and PSCs on HTTPS with port 5480
https://<appliancename>:5480
Step 2 – Configure Syslog with the
following settings.
Common Log Level
Info
Remote Syslog Host
<vRLI-LoadBalancer-VIP>
Remote Syslog Port
6514
Remote Syslog Protocol
TLS
vCenter Server Appliance
Step 1 – SSH to the VCSA and open the following file /etc/rsyslog.conf for editing.
vi /etc/rsyslog.conf
Step 2 – Press [Insert] to put vi into insert mode and add following entry towards the top of the file at the bottom of the ###### Module declarations ###### section.
$ModLoad imfile
Step 3 – Add the following right below the “###### Rule declarations
######” section of the rsyslog configuration file
$InputFileName specifies the log file that we want to forward.
$InputFileTag is the appname that will show up when it is forwarded to your remote syslog server
$InputFileStateFile is the log monitoring file.
$InputFilePollInterval is set 20 seconds, the default is 10 if you leave it blank.
Step 4 – Save your changes by pressing [Esc] and typing and pressing enter:
:wq
Step 5 – Restart the rsyslog service in the VCSA for the changes to go into effect by running the following command:
systemctl restart rsyslog
Platform Services Controller Appliance
Step 1 – SSH to the PSC and open the following file /etc/rsyslog.conf for editing.
vi /etc/rsyslog.conf
Step 2 – Press [Insert] to put vi into insert mode and add following entry towards the top of the file at the bottom of the ###### Module declarations ###### section.
$ModLoad imfile
Step 3 – Add the following right below the “###### Rule declarations ######” section of the rsyslog configuration file
