Hello,
We are running ESXi 6.7 U1 (13004448) and VCSA 6.7 U1b.
A few days ago I noticed our regular firmware backups which run Get-VMHostFirmware were not getting executed. In the hostd log, I saw the error esx.problem.visorfs.ramdisk.full and noted ams-bbUsg.txt taking up pretty much all of the space in /tmp. After removing that file, everything worked fine again. We did not get any notifications or snmp traps the /tmp directory was full.
To my question: Is it possible to create a custom alarm in vCenter to fire on this event? I tried going into vCenter => Alarm Definitions => Add => Target type = Hosts, and then simply entering esx.problem.visorfs.ramdisk.full in the Alarm rule, but I can't enter the full string into the textbox. I'm probably doing something wrong!
I'm attaching a picture of how I'm trying to add the custom alarm for clarity, and below is the full error from the hostd log.
Thanks in advance.
/Gustav Andersson
2020-04-24T21:42:43.852Z info hostd[2104254] [Originator@6876 sub=Vimsvc.TaskManager opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Created : haTask--vim.host.FirmwareSystem.backupConfiguration-6348273
2020-04-24T21:42:43.854Z info hostd[2104254] [Originator@6876 sub=SysCommandPosix opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] ForkExec(/sbin/firmwareConfig.sh) 6249455
2020-04-24T21:42:44.095Z info hostd[2099846] [Originator@6876 sub=Hostsvc.VmkVprobSource] VmkVprobSource::Post event: (vim.event.EventEx) {
--> key = 111,
--> chainId = -1,
--> createdTime = "1970-01-01T00:00:00Z",
--> userName = "",
--> datacenter = (vim.event.DatacenterEventArgument) null,
--> computeResource = (vim.event.ComputeResourceEventArgument) null,
--> host = (vim.event.HostEventArgument) {
--> name = "10-0-xxx-xx.local.xxx.xx",
--> host = 'vim.HostSystem:ha-host'
--> },
--> vm = (vim.event.VmEventArgument) null,
--> ds = (vim.event.DatastoreEventArgument) null,
--> net = (vim.event.NetworkEventArgument) null,
--> dvs = (vim.event.DvsEventArgument) null,
--> fullFormattedMessage = <unset>,
--> changeTag = <unset>,
--> eventTypeId = "esx.problem.visorfs.ramdisk.full",
--> severity = <unset>,
--> message = <unset>,
--> arguments = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "1",
--> value = "tmp"
--> },
--> (vmodl.KeyAnyValue) {
--> key = "2",
--> value = "/tmp/firmware-backup.6249455.lck"
--> }
--> ],
--> objectId = "ha-host",
--> objectType = "vim.HostSystem",
--> objectName = <unset>,
--> fault = (vmodl.MethodFault) null
--> }
2020-04-24T21:42:44.096Z info hostd[2099846] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 45621 : The ramdisk 'tmp' is full. As a result, the file /tmp/firmware-backup.6249455.lck could not be written.
2020-04-24T21:43:44.133Z error hostd[2104254] [Originator@6876 sub=Default opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] SysCmd [/sbin/firmwareConfig.sh --backup /usr/lib/vmware/hostd/docroot/downloads/524e04be-4d82-4d9e-03ea-db4dfb052267] failed with exit code [1] and output:
--> lockfile: Sorry, giving up on "/tmp/firmware-backup.6249455.lck"
-->
2020-04-24T21:43:44.134Z info hostd[2104254] [Originator@6876 sub=Default opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] AdapterServer caught exception: N5Vmomi5Fault11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
--> )
--> [context]zKq7AVICAgAAAKBuxgAOaG9zdGQAAEy4NWxpYnZtYWNvcmUuc28AAFDKGwCwuhcBg/ViaG9zdGQAAek2ZQFXy3aCEq0nAWxpYnZpbS10eXBlcy5zbwABWQvCAVEUwgD1TigA01IoAAsuNgM7fQBsaWJwdGhyZWFkLnNvLjAABB2bDmxpYmMuc28uNgA=[/context]
2020-04-24T21:43:44.138Z info hostd[2104254] [Originator@6876 sub=Vimsvc.TaskManager opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] Task Completed : haTask--vim.host.FirmwareSystem.backupConfiguration-6348273 Status error
2020-04-24T21:43:44.138Z info hostd[2104254] [Originator@6876 sub=Solo.Vmomi opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] Activation [N5Vmomi10ActivationE:0x0000006f2a5f19b0] : Invoke done [backupConfiguration] on [vim.host.FirmwareSystem:ha-firmwareSystem]
2020-04-24T21:43:44.138Z info hostd[2104254] [Originator@6876 sub=Solo.Vmomi opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] Throw vmodl.fault.SystemError
2020-04-24T21:43:44.138Z info hostd[2104254] [Originator@6876 sub=Solo.Vmomi opID=1800d84d-1f-6df3 user=vpxuser:VSPHERE.LOCAL\Administrator] Result:
--> (vmodl.fault.SystemError) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> reason = "Internal error"
--> msg = ""
--> }