mercredi 3 juillet 2013

Mod_JK : session mixed between users



I worked a few weeks ago on a particularly vicious bug.

Some users of our JEE web applications experimented a change of identity when they were working quietly.

Better, they were in the session of another user, set in a context that was not theirs.

Our architecture is based on Apache, Mod_JK and Glassfish.

A track


After googling a few minutes I found a serious lead: a critical bug in Mod_JK still open, entitled Response mixed betweens users.

The bug is opened in 1.2.28 as 1.2.19 but we must never forget that the discovery of a bug in a version that does not mean the bug did not exist before.

Even if the form does not mention the bug Glassfish, I assume this is a Mod_JK  bug.

So I decided to set up a test environment to try to reproduce the bug at will.

For this, some system commands later, I find myself with a proper environment.

Session mixing: how is this possible?

How to verify that occurs ?

If the bug is rare  then there is little chance that manual testing can reproduce the problem.

We must first understand how it happens : when a user receives the response to a query it was not intended to !

Indeed, the response header Set-Cookie is returned for each response sent by the Glassfish server. Note that Tomcat does not work the same way by issuing the response header Set-Cookie only once, when creating the session.

Glassfish and is more vulnerable to this Mod_JK  bug as all responses from the server contain this header field when the JVM is configured with a JVM route.

If Mod_JK mixes responses then users will receive a session cookie that is not for them.

How to check this?

After reading the comments on the bug, I've realized that this bug was kind of stealth or very stealthy.

The probability to reproduce this bug by hand,  playing with multiple browsers is almost zero.

JMeter to the rescue


JMeter is in my opinion a first-rate tool, both simple and powerful for those who know to exploit it (it's quite convenient for small load because of its thread based model).

It will allow me to generate a significant number of user sessions on a single JEE web and check for each query if the associated response contains the same Set-Cookie header.

We need a BeanShell assertion to check the crucial point: is there sharing session?

Here is the script (this is not a Groovy file but a BSH one, Gist does not know BSH).

You can find the JMeter script and BSH script here in my github repo.

This script should be used for each query in a BeanShell assertion with JMeter 2.9.

You can either reference a file that contains the code or paste the code in the box provided in the statement.

With this script I could reproduce the problem : check JMeter log file to see the mixing.

The rate is about 0.8 mixing for 1000 requests.

This is low, but for a web application that knows a sustained traffic, it is enough to be a major security risk.

Is this really a bug Mod_JK ?

Using Mod_proxy removed the behavior.

This does not come from the application. This observation is consistent with some evidence related to the bug.

Lessons


This expertise has been a confirmation of felt about the evolution of our architecture for me.

Several people were mobilized for a month to try to isolate and reproduce the problem, without success.

Architectures are more and more complex: it is almost impossible to solve problems without method or extreme rigor.

However, some professionals continue to understand the technical issues such as "before" no methodology is well established, no reliable configuration management of the architecture components, no trace when trying to copy, modify several parameters of architecture without properly control the scope of these changes, ...

A few days ago Michael Shallop published an article on the 5 best rules for debugging and diagnostic.

I recommend this article.




Contrat Creative Commons
the jee architect cookbook by Olivier SCHMITT est mis à disposition selon les termes de la licence Creative Commons Paternité - Pas d'Utilisation Commerciale - Pas de Modification 3.0 Unported.

4 commentaires:

  1. Can you post the JMeter test? The JMX file. I would like to know more details about recreating this issue.
    What kind of concurrency is necessary to start exhibiting the bug?

    RépondreSupprimer
    Réponses
    1. Yes i can post the Jmeter script, i must find it.
      I'd tested my script on POC app doing very little : just showing a simple page.
      I'd to try many times to get the bug, increasing concurrent users for every try : 10, 50, 100, 150, 200.
      But the bug was very easy to exhibit on a real application : concurrency is higher when application is doing something.
      I come back to you when i find the script.

      Supprimer
    2. Hello! We have a similar bug at https://github.com/IQSS/dataverse/issues/647 and are interested in seeing more of the script so we can reproduce the issue. Can you please post it?

      Supprimer
    3. Hello,

      scripts are here :
      https://github.com/olivier-schmitt/the-jee-architect-cookbook/tree/master/misc/tomcat/mixed-session

      Supprimer