Details


We were requested to perform a load test on our Shibboleth IdP server.

Earlier we had 2 GB ram and 2 cores but on load testing the concurrent users it could serve was around 355 users after which we started receiving socket timeout errors.

Increasing the Ram to 8 GB and Processor to 8 cores provided the same result with a small variation in page response times.

During the load testing, we observed that the individual CPU core utilization remains less than 40 percent and does not even consume 50% of the heap memory.



Cause


The logging level in DEBUG mode causes severe Disk IO waits, limiting the jetty process from serving more requests concurrently.



Recommendations


Please ensure that your Shibboleth IdP is not logging in DEBUG mode. Using DEBUG logging in production severely degrades IdP performance and results in terrible results during load testing.


To check /change the logging for your IdP,  edit the file in /opt/shibboleth/shibboleth-idp/current/conf/logback.xml and change the lines as below.

 <!-- Logs IdP, but not OpenSAML, messages -->
  <logger name="net.shibboleth.idp" level="INFO"/>

  <!-- Logs OpenSAML, but not IdP, messages -->
  <logger name="org.opensaml.saml" level="INFO"/>

  <!-- Logs LDAP related messages -->
  <logger name="org.ldaptive" level="WARN"/>



INFO is the recommended logging level for production servers.


Note: Log level changes will require an IdP restart.