karniv0re's Journal: Autonomous Love Hate

We use Autonomy's IDOL search engine extensively at work and have become sort-of experts on it. Well, to a point. See, Autonomy don't give out help and documentation all willy-nilly. You've got to pester the shit out of them to get anything useful.

Their documentation is shit. There's like, a 500 page PDF of server operations, but good luck with developer documentation. There's not so much as Javadocs for their API.

During this whole process, we've run into capacity issues. IDOL was taking around 16 seconds per index transaction.

I could see an abundance of connections on the Autonomy server in the status of FIN_WAIT2. This means the transaction has completed and it is just waiting for the client (our message processor) to close the connection. We are using Autonomy's API to index data, and the API itself is taking around 16 seconds to complete a connection, while the actual index transaction really only takes a few milliseconds.

I decompiled their code to see that they aren't releasing the connection when they should be which I believe is causing this FIN_WAIT2 problem.

According to the HttpClient documentation:

"Release the Connection: This is a crucial step to keep things flowing. We must tell HttpClient that we are done with the connection and that it can now be reused. Without doing this HttpClient will wait indefinitely for a connection to free up so that it can be reused.
method.releaseConnection();"

Using the exelent Java Decompiler by Emmanuel Dupuy, I was able to see exactly where they weren't calling method.releaseConnection(), and I pointed it out to them. Lawyered.

So now we're just waiting for them to admit they fucked up and give us an updated version of the API.

In the mean time, I talked with our server team to have them decrease the timeout on FIN_WAIT2s from 60 seconds to 3 seconds in /etc/sysctl.conf. I now no longer see the FIN_WAIT2s hanging out there. However, the connections are still taking around 16 seconds to complete, so I still think the API needs to be fixed to release the connection.

I'm wondering if there's a super-hack method to force the connections to close. I don't have access to the HttpClient library, because it is wrapped inside the ACI API, and that is just a JAR, no source. I'm going to do some digging though and see if I can force that to close. Last resort, if they won't fix this for us, is to write my own indexer client. Can't be terribly difficult. I have extensive experience with HttpClient from my previous project. Dare me to do it.