Preventing performance issues related to antivirus software
We get quite a number of support calls with either performance or system-down issues related to installation security suites, mostly related to antivirus software. In most cases the culprit ends up being the incorrect setup of the antivirus software.
Let’s first consider what antivirus software has to do and how it ships by default.
In today’s cat and mouse game, the security software vendors are trying to keep up with all of the malware generators that pop up daily. A typical antivirus signature file contains over 80 Mb of compressed signatures, and the major players like Trend, McAphee, Symantec, VIPRE, and Kaspersky provide multiple updates to signatures daily. The problem then is deciding what to scan and when to scan—you obviously don’t want to miss an infected file that’s downloaded between updates to the scan databases, but you also don’t want to bog down your system unnecessarily. By default, most security products scan all files once daily, and use real time scanning to scan infectable files on both read and write. Some even default to continuously scanning all files. Though each vendor has different terminology for “scan on read” and “scan on write” (in fact some confuse read as write and write as read), “scan on read” effectively means scan every time a file is opened and “scan on write” effectively means only scan when a file opened for write is closed. Some vendors even have a flag to scan all files on close. And some products, like VIPRE, don’t have any concept of scan on write only.
Now that we know how these products handle file access, let’s consider some scenarios on live systems.
Scenario 1 – When “scan all files” is set
In this scenario, every file may be scanned for a virus on open and close, regardless of writeability. Consider scanning a .vhd for a virtual image, or a Synergy DBMS file every time a user opens or closes the file. (Both file types are usually opened even for write.) The same would even apply to every file accessed in your SQL Server and Oracle databases, and to all of your Synergy .dbr and .elb files. The implications to your system performance are obvious.
Scenario 2 – Scan only infectable files
In this scenario, infectable files may be scanned on open and close. By default in most vendors’ products, this includes Synergy .ism files as well as .vhd files. This scenario as well has a significant impact on your system performance due to the overhead of scanning large files.
Scenario 3 – Scan only infectable files on Write
In this case, .exe and .dll files are only scanned when updated, but a .vhd and a Synergy .ISM file would also be scanned on close because they are usually opened for write. This technique might be good for a general purpose file server of Word documents, for example, but not for a data server.
As you can see, without some degree of tuning, virus scanning products can have disastrous effects on system performance. (You can use the Sysinternals Process Monitor to see the overhead your virus scanning tool is causing.)
For obvious reasons, scanning of files takes place at a high priority in the kernel mode of the operating system. This usually impacts both system time and user processing time. Additionally, many vendors now use the VISTA filter manager, and I previously bloggedabout the performance penalties of such hooking on Vista and Server 2008. Luckily the overhead is significantly reduced in Server 2008 R2 and Windows 7.
In our recent internal use of Microsoft’s SharePoint server, we were seeing dramatic performance problems when installing and uninstalling software, and even when the IIS SharePoint services (which are .NET-based) were loading and jitting. By correctly disabling the “scan on open for read” options, the performance significantly improved. We also tried the VIPRE product, and this improved performance even further – however, for a very specific reason. VIPRE, as stated previously, scans all files on open and close, and gains its performance edge because it recognizes signed, read-only EXE/DLL files and caches them if they have not changed so that the re-scan is not required. This is what gives it a seemingly large performance gain. However, once you throw in files that are not signed, its scan requires significantly more resources because you can’t disable the “scan on read” functionality (which would require a scan of such products as Diskeeper moving around files). Additionally, VIPRE also scans (but does not report issues with) other excluded files, so the overhead is pretty much permanent for unversioned files like Synergy DBMS files.
The key is, after you have a clean full-file scan on a system, set scan on write only, scan infectable files, and make sure that the file extensions of your databases and VHD files are set to no scan. And, due to its inability to scan on read, we do not recommend VIPRE for use with Synergy/DE installations.
(Of course I’m providing this information for information purposes only, and it is up to each company to set its security policies.)
Protecting the Spread of Security Infections in Places You Might Not Think About
Several weeks ago we had a new Ikon color printer installed. It has a separate Kodak PC running the printer drivers and color matching software. I noticed that it was Internet connected and that software updates were not being applied.
When we contacted the manufacturer, we were told the PC was an embedded XP device and did not need the XP SP3 nor the security patches. We immediately disabled the Internet connection (embedded XP devices are susceptible to viruses too)—but that’s not really good enough. To date the manufacturer still has not authorized XP SP3 nor the regular monthly security patches, yet all printed documents go through this machine and users can go to the console and copy documents from a USB drive or internal network locations. Once infected with a virus or worm — or even a botnet — we’re SOL, because the manufacturer of the device doesn’t support installing anti-virus software, and any such changes would require an engineer to reload the system from scratch.
The problems are not just with Microsoft. Adobe has had to patch its Flash Player and Reader already this year, and another Reader patch is due. How many of us keep the Adobe Reader and Flash players up to date?
Why is this such a big issue? Well, the problem is that these embedded XP systems can get infected. One example is the Conficker worm. In most cases Conficker is benign until it is woken up by its creators. Users don’t even know they have it, may not even have Internet access (or may not know that they do), and/or may have been infected internally. The only way to detect these kinds of issues other than with a virus scanner is to look at network traffic going back to “phone home.” I think an article from the San Jose Mercury News illustrates the problem well. Even if you have a patch available to avoid infecting a machine, what if every patch and/or daily antivirus update required a 90-day approval process?
My recommendation is that you get with the manufacturers of all embedded XP devices that are connected to your network and get the regular updates and XP SP3, and ensure that Internet Explorer is disabled in such a way that the machine’s users cannot re-enable it. And also be sure to keep your Adobe Reader, Flash players, and similar products up-to-date.
The Vista performance saga – final chapter
In January we finally determined why file I/O on Vista and Server 2008 disks is slower than on Windows 2003. In a previous blog post I stated that
“The performance problem on disks that have been hooked by applications that use the new Vista/Server2008 filter manager infrastructure – can cause CPU overheads of at least 40% on all I/O operations including cached I/O and locks reducing throughput.”
So what applications use the new filter manager? Well UAC on system disks using the UAFv.sys file system re-director use the filter manager, and many current antivirus applications use the filter manager on all the disks where they are set to perform real-time scanning.
In Vista the initial hit is high to register “any” application to use the filter manager on a volume and then rises even higher for every operation type hooked. The UAC file system re-director – that ensures that writes to Windows-protected directories like windows\system32 and \program files\ are re-directed to the user’s local path, which the user does have access to. If you use Yahoo Messenger on a Vista system, you will see it has this problem because it always assumes it can write to program files. Now the reason that the uafs.sys file system redirector hooks every file I/O operation on the system disk is because it tries to cache these re-directed operations to avoid creating and writing the temporary re-directed file to disk ever; however this now causes the performance issue on Vista unless file system redirection is turned off by disabling the service (which may cause applications like Yahoo Messenger to fail unless UAC is also turned off).
I had turned uafv.sys off on my Vista system – however performance traces in Intel’s VTUNE performance advisor showed that I was still getting performance degradation due to the filter manager when running our test suites. It turns out that the latest Trend Micro antivirus engine is following Microsoft’s best practices and using the new filter manager on all disks – so the previous work-around of using a non system disk did not work on my machine.
In my dialogue with Microsoft, they indicated that they did not expect the data drives of an internal file server to always need to have an antivirus scan (by this I don’t mean a file server in the Word document sense, rather a dedicated database server that has no internet access), so the overheads related to the virus scanner would not apply to non system disks – and even if a virus scanner was installed that it would only be set to scan the system disk in real-time mode.
The good news is that Windows 7/Server 2008 R2 have significantly improved this situation. Though there is some overhead for the initial attach to the filter manager, additional attaches cause much less overhead, and the overall figure is far better than Vista. Microsoft will continue to look at this area during the release cycle of Server 2008 because of the impact it has when virus scanners are using the filter manager and set to real-time scan all disks on a system.
Microsoft’s ADO.NET Entity Framework
Over the years, Microsoft has provided many different ways to access data–ODBC, DAO, ADO, and ADO.NET (with data sets and data readers). The next data access technology is the Entity Framework with the 3.5 SP1 version of ADO.NET. Synergex has provided access to all of these technologies through the baseline ADO.NET 2.0 with its xfODBC driver. Synergex has developed its own ADO.NET 3.5 provider with the extended capabilities needed to interoperate with the Entity Framework and the Entity designers in Visual Studio 2008 SP1.
Microsoft views the Entity Framework as the future of all of its data access technologies – and products like SQL Server, Office, and the Visual Studio designers are all either upgraded or being upgraded to require access to databases via the Entity Framework.
Here is how Microsoft describes the ADO.NET Entity Framework:
“Database development with the .NET framework has not changed a lot since its first release. Many of us usually start by designing our database tables and their relationships and then creating classes in our application to emulate them as closely as possible in a set of Business Classes or (false) "Entity" Classes, and then working with them in our ADO.NET code. However, this process has always been an approximation and has involved a lot of groundwork.
This is where the ADO.NET Entity Framework comes in; it allows you to deal with the (true) entities represented in the database in your application code by abstracting the groundwork and maintenance code work away from you. A very crude description of the ADO.NET Entity Framework would be that It allows you to deal with database concepts in your code.“
The ADO.NET Entity Framework is designed to enable developers to create data access applications by programming against a conceptual application model instead of programming directly against a relational storage schema. The goal is to decrease the amount of code and maintenance required for data-oriented applications. Entity Framework applications provide the following benefits:
-
Applications can work in terms of a more application-centric conceptual model, including types with inheritance, complex members, and relationships.
-
Applications are freed from hard-coded dependencies on a particular data engine or storage schema.
-
Mappings between the conceptual model and the storage-specific schema can change without changing the application code.
-
Developers can work with a consistent application object model that can be mapped to various storage schemas, possibly implemented in different database management systems.
-
Multiple conceptual models can be mapped to a single storage schema.
If you are interested in beta testing our new Entity Framework capabilities, please contactSynergy/DE Developer Support.
For more information and a tutorial of the Entity Framework, see these links:
http://msdn.microsoft.com/en-us/library/aa697427(VS.80).aspx
http://www.codeguru.com/csharp/csharp/cs_linq/article.php/c15489/#more
In: Uncategorized
Upcoming “experimental feature” will help you detect use of uninitialized memory
We are continually reviewing customer applications to assist with support/development issues, and in doing so often come up with ideas to help customers facilitate debugging problems they may encounter. We use a product from Compuware called DevPartner Studio to help us track down “C” variable access problems in the Synergy components that sometimes cause instability in the runtime. I like to run customer applications with a special runtime that is built with DevPartner, which allows us to check boundary conditions while running “real” customer applications. DevPartner enables us to check use of memory already freed (called dangling pointers) and access to memory before we have written to it (a common cause of symptoms that move around depending on memory and time of day).
One recent application we saw was accessing uninitialized memory before writing to it. As we tracked this down, , we realized the customer was using stack records and %MEMPROC memory that had never been written to. In certain cases this would cause random results, and in this particular case, it was causing the customer’s application to fail when run under the DevPartner tool because the memory was now a consistent but unexpected value.
We decided as a test to add some support in Synergy/DE to see if the Synergy runtime could also detect this use of uninitialized memory with a minimal overhead when running in debug. It turns out that we can do similar checking for assignment statements and “if” tests, and we can differentiate between stack memory and MEM_PROC memory. Using this functionality also enables a developer to break in the debugger after the statement that uses this random memory.
We are considering adding this new debugging functionality to a future release of Synergy/DE. However, so that we can get this useful tool into your hands sooner, we are planning to include it as an “experimental feature” in an upcoming patch.
“Experimental features” are features that are under evaluation. They are for early adopters to use and provide us with feedback on. They will be supported, but they may be modified or even removed in subsequent releases.
So look for this new experimental debugging feature in an upcoming patch and consider trying it out. Like the recent feature we added to detect mismatched global data-section sizes (which can cause runtime crashes), this feature to detect uninitialized memory continues our aim to add debug-time detection of coding errors to assist you in producing more reliable applications.
In: Uncategorized
Live from Microsoft PDC: A sneak peak at Windows 7, plus our 64-bit ActiveX list support
This comes to you from the Microsoft PDC in Los Angeles, where I am among over 10,000 attendees. The PDC is Microsoft’s futures conference where they preview some of the technology coming out over the next couple of years.
Microsoft has demonstrated real UI improvements in Windows 7—improvements that made almost every attendee cheer. For example, Windows 7 includes UAC improvements so you don’t have to accept “On” or “Off”. And, the new iPhone-like touch support is certainly cool. It looks like within 5 years almost every laptop and LCD monitor will include touch support. The great thing with touch is that there are no UI Toolkit changes required to your Synergy/DE Windows applications because touch translates to normal mouse movements and clicks.
Microsoft has also set a goal to make Windows 7 run faster, boot faster and require less memory than Vista, targeting the new ultra mobile 10" laptops that have flash drives and 1GB of memory. This goes hand in hand with new features in the .NET framework that reduce memory requirements and provide improved interoperability with lower overheads. At Synergex we will be testing Synergy/DE with Windows 7 in the near future—to ensure everything works as well in Windows 7 as it currently does in Vista and Server 2008. Windows 7 also contains the same set of files as Server 2008 R2 so any performance improvements in Windows 7 will also benefit the server platform.
I also want to let you know that we have recently completed our 64-bit ActiveX list implementation, and it will be released in our upcoming 9.1.5a version. This means that 64-bit UI Toolkit applications are now possible on 64-bit native operating systems with the same features as their 32-bit counterparts (that is, if the appropriate controls you use are also available). This now enables you to take full advantage of the extra memory and scalability available with Server 2008 x64 Edition. (Server 2008 R2 is already announced as the last 32-bit server O/S by Microsoft.)
The Vista Performance Saga Continues
I thought it about time I posted an update regarding my Vista post on the 16th of April. In that post I recommended holding off on Server 2008 deployments until more data was available.
So let’s state the real problem.
“All file operations (read, write, file-position, etc.) are 40% slower on a Vista and Server 2008 system disk than they are on XP or Server 2003 system disks.”
These operations are slowed down even when they are serviced from the O/S cache subsystem. The reason for the 40% overhead is the registration of a driver with the newly (Vista) introduced file system filter framework, even if the driver itself performs no work and just returns. Registration can be for a particular device and not just a disk drive. In one case, the UAC file system virtualization driver, UAFV.SYS registers itself with the filter manager framework to perform the protected file virtualization feature new in Vista. As a result of the filter manager subsystem overhead – all read/write/seek operations to the C: drive become slower regardless of the file virtualization operation. Turning off this UAFV.SYS driver restores system disk performance.
How can you tell what this means? You can use the sysinternals procmon utility to see all the I/O operations occurring on your c: drive—every one of those operations is slowed down on a Vista and Server 2008 system disk. This accounts for some of the CPU bottleneck when your laptop starts. It accounts for slower virus scans on Vista system disks, etc.
As nearly all laptops, most small business servers, and the majority of current desktops all have a single system disk, this problem impacts all current Vista and Small Business Server 2008 users to some degree or another. This problem becomes exacerbated when other utility and anti-virus software takes advantage of the new Vista filter manager framework, where performance to non-system disks will be impacted.
Solutions are of course to read/write sequential data in much larger blocks. We changed Synergy/DE to use 4k buffers for sequential output in our recent 9.1.5 release, however the semantics of the sequential input read allowing for random reading precludes us from doing that on input without slowing down performance. Random ISAM reads can’t use larger blocks without damaging performance at the disk level—so they incur the CPU overhead. Most of the I/O patterns I see with procmon also don’t meet the bar for larger I/Os, so the real issue is to get the problem fixed in the O/S.
If you disable UAC (which we don’t recommend) and you have never virtualized a file (for example, you do this at system installation), you can use the registry editor to make the uafv.sys service visible and then disable it. Doing so will also mean you can’t re-enable UAC till the service is re-enabled. Alternatively you can ensure all your data files (this also means your temp and DTKTMP logicals) are placed on a non system drive – and you won’t see most of the impact of this problem.
We are currently working with Microsoft to provide a fix to this in the next Service Pak and and will keep you informed of our results.
As a side note, we also noticed that any scheduled task runs slower in Vista and Server 2008. Typically customers use these to generate reports and run day ends overnight. These tasks now run at a low priority class. You would expect an idle system to run them almost the same—regardless of the priority class (after all the idea is low priority items use available resource when there are no higher priority items running), but it appears that the programs no longer use available resources as prior versions do. Microsoft sees this as by design—which is hard to believe. We have introduced a new API in 9.1.5 to allow you to re-set the priority class of your scheduled tasks to ensure they retain the performance characteristics of prior operating system versions.
In: Uncategorized
Red Alert! DNS Flaw Revealed
Due to the recent online disclosure of technical details to exploit a widespread DNS vulnerability, security researchers are warning users to patch vulnerable systems immediately.
All Linux and Windows based DNS servers require a patch, and most routers need a patch with real urgency.
From InformationWeek.com:
The domain name system translates domain names, like "informationweek.com," into numeric IP addresses and vice versa. The DNS flaw, if exploited, allows what is known as DNS cache poisoning. This involves remapping domain names to different, potentially malicious servers.
US-CERT on Monday warned: "Technical details regarding this vulnerability have been posted to public Web sites. Attackers could use these details to construct exploit code. Users are encouraged to patch vulnerable systems immediately."
"This is a very serious situation, and can possibly lead to widespread and targeted attacks which hijack sensitive information by redirecting legitimate traffic to fraudulent Web sites, due to incorrect (fraudulent) information being injected into the vulnerable caching nameserver(s)," Trend Micro security researcher Paul Ferguson said in a blog post.
Read the full article:http://www.informationweek.com/news/internet/security/showArticle.jhtml?articleID=209401195
For additional information about this type of attack and for details about how to resolve it, visit http://www.kb.cert.org/vuls/id/800113.
The XP era is over – what does that mean to you?
As Windows XP is no longer available as of June 30th, I’d like to talk about your options regarding Synergy/DE support for Windows Vista.
While Microsoft may have pulled the plug on Windows XP as of June 30, it still continues to offer the home version for ultra low end PCs that can’t run Vista. However, if you go to Dell or HP, you won’t be able to select XP for a new system. Manufacturers can continue to sell XP while “stocks last” but in today’s highly evolving marketplace, who would stock XP just in case someone might buy it one day? Further, volume license customers can’t purchase XP licenses any more—the only way for a business customer to get it is to buy Vista Enterprise and downgrade to XP.
So, where does that leave Synergy/DE customers sitting on the fence and using versions of Synergy/DE prior to 9? Well, as of July 1, the supported route is to upgrade to version 9. Any new machines your customers/users buy will be running Vista, which means you need version 9 for that user (if you want to deploy a supported version). We just shipped our latest version in the 9 series, version 9.1.5, which we recommend using.
So what do you do if you want to use Vista and Server 2008 but your installed base is using 8.3.1# and you don’t want to upgrade them all at once? We have customers who have been accomplishing all of this successfully by continuing to build their .dbr and .elb files with 8.3 and then running those 8.3-built files under Synergy/DE 9. In the rare documented cases where version 9 finds an issue not present in 8.3 (e.g., the new duplicate global data section of differing sizes), the issue can be fixed back in the 8.3 code base producing a .dbr that runs perfectly on both 8.3 and 9. This same technique should be used if you are requiring a hotfix for a problem in 8.3. Synergex’s policy is to provide Synergy/DE 9 for deployments of the fix rather than an 8.3 patch.
Now you may ask, what about development? We still recommend you use the latest version 9 tools to build and develop your applications (so you can take advantage of improved error detection and increased developer productivity), but you can rebuild the tested .dbr files under 8.3 for mass deployment.
Given that the XP era has ended, I recommend that all ISVs test their current pre-9 applications under Synergy/DE 9.1.5 so they can be assured of continued customer satisfaction when the inevitable Vista machine is encountered. I also recommend that all new customer installations be V9 throughout, or at least adopt the built-under-8.3-deployed-under 9 model described above.
Don’t forget support for your non-Synergy/DE products
In my last post, I talked about some issues with Windows Server 2008 and Vista SP1 that caused me to recommend not upgrading to them yet. These issues represent just one example where an operating system problem might hinder performance for our customers.
In another example, we recently had a customer report that it was taking our SQL OpenNet server 20 times longer to retrieve records from a SCO OpenServer 6 or UnixWare system than from SCO OpenServer 5.0.6. We tracked this down to a bug in the SCO implementation of the Nagle algorithm on the TCP/IP stack. We produced a simple C program that was sent to SCO and a fix is pending.
While we were able to assist the customer in the above situation, this isn’t always the case. We try hard to reproduce operating system and other layered product problems with our support team even when Synergy/DE is not at fault, but we unfortunately can’t support every OS and product in the field. There is an increasing need for our ISVs and end customers to maintain software support contracts with the vendors they work with to solve problems.
In many cases the problems we come across are third-party interaction issues (like virus scanners) and configuration issues with the OS that are beyond the scope of Synergy/DE support. A prime example of this is the use of operating system virtualization, where Synergy/DE is supported on the target OS, and the virtualization software acts as a hardware layer underneath the OS. As we have found out, Microsoft will not entertain any calls being logged if the problem is not reproducible in a non-virtual environment. So just as the device drivers of a server require a maintenance contract with the hardware supplier, so the use of virtualization software requires the same (effectively hardware) support contract with the virtualization supplier.
So I recommend you evaluate the level of support you may need for your non-Synergy/DE products and then obtain the appropriate support contracts.
