- Microsoft acknowledges the issue in Exchange 2016 &2019 that leads to Email Stuck in Transport Queues.
- Microsoft with the official fix and guide on how to apply.
The root cause of the bug was due to using a signed int32 for the date and the new date value by Microsoft, of 2,201,010,001 is overvalued the maximum value of a signed long integer i.e 2147483647. The bug gets triggered when the “MS Filtering Engine Update” process received the “220101001” update version just some hours before the New Year, and the process started crashing as time was 0000 (O'clock). In the application event logs, the errors from the FIPFS service say: Cannot convert “220101001” too long.
This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.We are actively working on resolving this issue and expect to release details on how to resolve this issue later today. In the meantime, if your organization performs malware scanning of messages outside of your on-premises Exchange servers (for example, by routing mail through Exchange Online, or by using a third-party message hygiene solution), you can bypass or disable malware scanning on your Exchange servers and clear your transport queues.
Microsoft assures the users that they are working on a fix of the issue and an update is in final test validation. This update requires customer action, but it will provide the quickest time to resolve the issue for users.
In the meantime, Microsoft recommends Two documents be available on this topic (how to disable or bypass malware scanning). One is about the Antimalware protection in Exchange Server, while the other covers procedures for antimalware protection in Exchange Server.
According to the Reddit thread, these bad updates affected Microsoft Exchange 2016 & 2019 versions. Furthermore, a Reddit user @ceno666 also confirms that the issue also seems to occur with the 220101002 update version as well.
Note this is only for On-Prem routed mail it seems and of course, 365 is working correctly
Solution and Guide to Apply Fix
Using the Automated Solution
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1 EXCH1 Stopping services... EXCH1 Removing Microsoft engine folder... EXCH1 Emptying metadata folder... EXCH1 Starting services... WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start... WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start... WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start... WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start... WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start... EXCH1 Starting engine update... Running as EXCH1-DOM\Administrator. -------- Connecting to EXCH1.CONTOSO.com. Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate -------- [PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation Engine : Microsoft LastChecked : 01/01/2022 08:58:22 PM -08:00 LastUpdated : 01/01/2022 08:58:31 PM -08:00 EngineVersion : 1.1.18800.4 SignatureVersion : 1.355.1227.0 SignatureDateTime : 01/01/2022 03:29:06 AM -08:00 UpdateVersion : 2112330001 UpdateStatus : UpdateAttemptSuccessful
Using the Manual Solution
Remove existing engine and metadata
- Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
- Use Task Manager to ensure that updateservice.exe is not running.
- Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
- Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.
Update to the latest engine
- Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
- Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.
Verify engine update info
- In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
- Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.
If you previously disabled or bypassed antimalware scanning as a mitigation for this issue, we recommend that you re-enable it after performing the above steps.
Fix for Exchange mail flow breaks
On the 10th of the December, Microsoft Exchange team already announced that team will not be releasing any Cumulative Updates (CUs) for Exchange Server in December and will share more information about our next CU release at a later time. So to date, there is no news for the update.
Security enthusiastic people calling this issue as Y2K22 and triggering this Time value bug on many 2016 & 2019 exchange servers, makes many server admins unhappy. Exchange Server admins can temporarily fix the issue by disabling anti-malware scanning, to restore mail flow. For this purpose, there is the script Disable-AntiMalwareScanning.ps1. Then malware scans are no longer executed
CD "C:\Program Files\Microsoft\Exchange Server\V15\Scripts\"
Another way is by disabling the filtering on the exchange server -
Set-MalwareFilteringServer exch-19 -BypassFiltering $true
Hope the Microsoft Exchange team soon released a fix of the bug.
Note: Disabling OR bypassing anti-malware filtering puts many servers at risk of phishing or other malware threats.
History of the Same bug (Y2K)
In the year 1999, a bug arose due to a problem in the coding of computerized systems that was projected to create havoc in computers and computer networks around the world at the beginning of the year 2000. This bug dubbed as Y2K (in metric measurements, k stands for 1,000), also called the Year 2000 bug or Millennium Bug,
Until the 1990s' many computer programs (especially those written in the early days of computers) were designed to abbreviate four-digit years as two digits to save memory space. These computers could recognize “98” as “1998” but would be unable to recognize “00” as “2000,” perhaps interpreting it to mean 1900.
Many feared that when the clocks struck midnight on January 1, 2000, many affected computers would be using an incorrect date and thus fail to operate properly unless the computers’ software was repaired or replaced before that date. Other computer programs that projected budgets or debts into the future could begin malfunctioning in 1999 when they made projections into 2000.
In addition, some computer software did not take into account that the year 2000 was a leap year. And even before the dawn of 2000, it was feared that some computers might fail on September 9, 1999 (9/9/99), because early programmers often used a series of 9s to indicate the end of a program.
In the year 1998, Microsoft issued a software fix to address the Year 2000 problems in Windows 98 that the company described as more of an inconvenience than a threat. Without the fix, certain parts of the operating system might inaccurately display dates come the millennium, Microsoft said.