EQuIS Enterprise 5 - Troublesh…
  • RSS Feed

Last modified on 9/27/2013 10:58 AM by User.

EQuIS Enterprise 5 - Troubleshooting - Performance

Keywords: Enterprise performance, EZView performance, slow, slowness, speed up, faster, quicker, internet information services, ISS

This article describes some ideas that typically help optimize performance of EQuIS Enterprise. Due to the variable nature of each deployment environment, we cannot guarantee any specific performance improvement for any of the ideas described herein. However, being familiar with the issues and implementing ideas applicable to your specific environment will usually result in noticeable performance improvements.

Many different variables affect the overall user-perceived performance of EQuIS Enterprise. Whenever the user interacts with EQuIS Enterprise, an entire chain of events occurs. Viewing a typical page within EQuIS Enterprise requires at least the following steps:

  1. The web browser (i.e. Internet Explorer) creates the HTTP request.
  2. The HTTP request is sent over the network to the web server.
  3. The web server (i.e. IIS (Internet Information Services)) receives the request and delegates processing to the ASP.NET Application Pool worker process (i.e. w3wp.exe).
  4. The worker process loads the EQuIS application, which begins processing the request.
  5. The EQuIS application establishes a connection to the EQuIS database.
  6. The database request is sent over the network to the database server.
  7. The database server (e.g. SQL Server or Oracle) receives and processes the request.
  8. The database response is sent over the network to the web server.
  9. EQuIS receives the database response, writes applicable information to log files, and creates the HTTP response.
  10. The HTTP response is sent, via ASP.NET and IIS, over the network to the web browser.
  11. The web browser receives the response, parses the response, and displays the page to the user.

These steps are often all completed within a few seconds or less and the user does not observe any delay. However, other scenarios exist where noticeable delays may occur. As illustrated in these steps, there are potentially three different machines (end-user workstation, web server, and database server), and two different networks (end-user to web server, web server to database server) that all work together for each user interaction. Any of these components can cause delays and should be considered when troubleshooting sub-par performance. It is often helpful to isolate specific components during the troubleshooting process. For example, the end-user workstation and the network between user and web server may be eliminated by using a web browser directly on the web server. This can be done by logging into the web server with Remote Desktop, opening Internet Explorer, and browsing to the local IIS instance hosting EQuIS Enterprise (e.g. http://localhost/equis).

Return to Top

Web Server Overload

One of the most common causes of poor performance is too many processes competing for resources on the web server. These steps will help to identify process(es) that may be competing with EQuIS and contributing to sub-par performance.

  1. Log on to the web server using Remote Desktop.
  2. Start Task Manager and, on the Processes tab, show processes for all users.
  3. Sort the processes by descending CPU usage (click on the CPU column so those processes using the most CPU appear at the top of the list).
  4. Either on the web server, or from a workstation, open Internet Explorer and start EQuIS Enterprise.
  5. As you use EQuIS Enterprise (login, view dashboard, run reports, etc.), watch the Task Manager and observe the processes that are using the most CPU and the most memory (sort by descending memory usage to determine which processes are using the most memory).
  6. Use the Performance tab to observe how much physical memory is in use and how much is available

There are several processes that may be running on the server that consume a lot of resources. Here are a few of the common types of applications that may need to be reviewed:

  • Antivirus/Antimalware Services (e.g. msmpeng.exe): Antivirus/Antimalware applications are a critical component of a safe computing environment. However, these components can also consume a lot of system resources and degrade the performance of other applications. For example, if an antivirus application is configured to continuously scan the entire file system, then scanning may affect performance as IIS/ASP.NET/EQuIS work with log files, temporary ASP.NET files, etc. Configuring the antivirus application to minimize conflict can improve performance of other applications on the server.
  • Content Indexing Services (e.g. cisvc.exe): Content indexing services scan files and folders on the server to support quicker searching. Unless you specifically need search capability on the server, we recommend disabling any context indexing services. If you need to maintain these services, they should be configured to do indexing when they will not conflict with other applications on the server.
  • Microsoft Sharepoint: Microsoft Sharepoint consists of several different processes and websites. We highly recommend that all Sharepoint components be installed on a separate server. Sharepoint is a resource intensive application that can significantly degrade the performance of other applications running on the same server. We particularly recommend that you disable the Sharepoint indexing component or configure it to index only during off-peak, low-usage periods on the server.
  • Microsoft SQL Server: Microsoft SQL Server consists of several different components. Some of those components may not be necessary, depending on how you are using SQL Server. For example, if you are not using SQL Server Browser, SQL Server Analysis Services, SQL Server Integration Services, or SQL Server Reporting Services, those services should be turned off to conserve system resources. 
  • Server Management Applications (e.g. momhost.exe): Server management applications, such as Microsoft Management Server, are valuable tools in managing large IT environments. However, these components can also consume a lot of system resources and degrade the performance of other applications. We recommend that you review any applications of this type and determine whether they can be turned on only when needed, or configured to minimize conflicts with other applications.
  • Other Services: We recommend that you review the Services console and understand each of the services running on the server. Any service that is not essential to desired server functionality should be disabled to maximize the system resources available to essential services.

If physical memory usage is high, and all non-essential services have been turned off or disabled, then you should consider increasing the memory (i.e. RAM) in the server to support the necessary services. 

Return to Top

Logging with log4net

EQuIS Enterprise uses the log4net logging component to log error messages and other information to log files (found in the .\EarthSoft\EQuIS Enterprise\bin\logs folder). Error messages and debug information is useful for troubleshooting specific problems that may occur. However, writing messages to the log file requires disk I/O and may trigger unnecessary scanning by antivirus or content indexing services. The log4net configuration file (.\EarthSoft\EQuIS Enterprise\bin\log4net.config) controls how much information is written to log files. Within the log4net.configuration file is a section that looks like this:

  <!-- what level of logging do you want to use:
      warn: only logs errors and warnings
      all: logs errors, warnings, and information messages (uses more disk space)
    -->
  <root>
    <level value="warn" />
    <appender-ref ref="RollingLogFileAppender" />
  </root>
</log4net>

The <root> logger element contains a <level> element. The value attribute of the <level> element controls the amount of information that is written to the log file(s) as explained here. The following values may be used to assign the level of logging:

  • ALL
  • DEBUG
  • INFO
  • WARN
  • ERROR
  • FATAL
  • OFF

These values are listed in the order of increasing priority, which means that any level automatically includes all levels below it. For example, the "warn" level will include warn, error, and fatal.

In order to minimize disk I/O and reduce performance overhead, we recommend that you reduce the logging level (e.g. "warn" or "error") when not troubleshooting a specific issue.

Return to Top

Location of log files

In some environments, ASP.NET will automatically attempt to recompile the web application when any files change within the application folder. The log files located in .\EQuIS Enterprise\bin\logs change frequently (particularly when the logging level is above "WARN"). The frequent recompilation may cause poor performance of the web application. If you notice the csc.exe process (C# Compiler) running on most or all web requests, then you should consider moving the location of your log files and work folder (see below). Moving the log files to a different directory (outside of the .\EQuIS Enterprise folder) will reduce the frequency of recompilation and improve overall performance of the web application. You can change the location of the log files by following these steps:

1. Stop IIS

2. Stop the EQuIS Enterprise Workflow Service

3. Browse to the .\EQuIS Enterprise\bin folder

4. Select the .\logs folder and move the folder (i.e. cut and paste) to a new location (e.g. C:\temp\EQuIS Enterprise\logs)

5. Set write/modify permissions on the new folder for the application pool identity user and the workfow service identity user

6. Open the .\EQuIS Enterprise\bin\log4net.config file

7. Find the section that sets the log file name (approximately line 10):

<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
    <param name="File" value="logs/{process}.log" />

8. Change the value to include the path to the new log folder (use forward slashes instead of back slashes):

 <appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
    <param name="File" value="C:/temp/EQuIS Enterprise/logs/{process}.log" />

9. Save log4net.config

10. Restart the EQuIS Enterprise Workflow Service

11. Restart IIS

Return to Top

Location of Work folder

As above, when documents are uploaded through Enterprise (through the web interface, or by email), ASP.Net may automatically attempt to recompile the web application due to perceived "changes" in the file/folder structure. The frequent recompilation may cause poor performance of the web application. If you notice the csc.exe process (C# compiler) running on most or all web requests, then you should consider moving the location of your work folder and log files. Moving these folders / files to a different directory (outside of the .\EQuIS Enterprise folder) will reduce the frequency of recompilation, and improve overall performance of the web application. You can change the location of the work folder by following these steps.  

1. Stop IIS

2. Stop the EQuIS Enterprise Workflow Service

3. Browse to the .\EQuIS Enterprise\bin folder

4. Select the .\work folder and move the folder (i.e. cut and paste) to a new location (e.g. C:\temp\EQuIS Enterprise\work)

5. Set write/modify permissions on the new folder for the application pool identity user and the workfow service identity user

6. Open the .\EQuIS Enterprise\bin\EarthSoft.Workflow.Service.exe.configuration file

7. Find the fileManagementConfiguration section that sets the log file name (approximately line 40-50), and add the attribute 'workFolder':

<!-- workFolder attribute is optional. It is the working folder where files will be stored during processing.  If not present, the default is ./bin/work -->  <fileManagementConfiguration workFolder="C:/temp/EQuIS Enterprise/work">

8. Save the EartSoft.Workflow.Service.exe.config file

9. Go up one folder to the EQuIS Enterprise root folder (.\EQuIS Enterprise), and open the web.config file

10. Find the line that sets the work folder (Approximately line 56-57):

<!-- This is the location of the working folder where files will be stored during processing (default is .\bin\work) -->
 <add key="workFolder" value="" />

11. Change the value to include the path to the new work folder (make sure to use the exact same path specified in step 7)

<!-- This is the location of the working folder where files will be stored during processing (default is .\bin\work) -->
 <add key="workFolder" value="C:/temp/EQuIS Enterprise/work" />

12. Save web.config

13. Restart the EQuIS Enterprise Workflow Service

14. Restart IIS

As mentioned previously, the most benefit will likely be realized when both the log files, and the work folders are moved at the same time. This will also only require stopping IIS and the workflow service once, instead of twice. If you're seeing performance issues due to the aforementioned recompilation issues, you may realize the greatest improvements by making both changes at once.

Return to Top

Database Connection Protocol

Microsoft SQL Server supports a few different database connection protocols. If the database server and the web server are the same machine (i.e. SQL Server and IIS are running on the same server), then the Shared Memory protocol is the fastest communications protocol. If applicable, make sure Shared Memory is configured correctly as explained here

Return to Top

IIS Web Gardens

Microsoft Internet Information Services (IIS) includes a feature known as "Web Gardens" that allow IIS to use multiple worker processes to support an application. As explained in this IIS blog post, web gardens should not be used in all scenarios:

"Web gardens was designed for one single reason – Offering applications that are not CPU-bound but execute long running requests, and the ability to scale and not use up all threads available in the worker process. ... [Examples may include] Applications that make long running database requests (e.g. high computational database transaction)."

EQuIS Enterprise, particularly EZView, often involves long running database requests, which makes it a potential candidate for benefiting from web gardens. Follow these instructions to enable web gardens for EQuIS Enterprise:

  1. In IIS Manager, expand the local computer, expand Application Pools, right-click the application pool, and then click Properties

  2. Click thePerformance tab, and under Web garden, in the Maximum number of worker processes box, type the number of worker processes that you want to assign to the application pool. (You must type a number greater than 1 for the application pool to become a Web garden).

  3. Click OK.

Return to Top