×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Autonomous Love Hate

karniv0re (746499) writes | about a year and a half ago

User Journal 0

We use Autonomy's IDOL search engine extensively at work and have become sort-of experts on it. Well, to a point. See, Autonomy don't give out help and documentation all willy-nilly. You've got to pester the shit out of them to get anything useful.

Their documentation is shit. There's like, a 500 page PDF of server operations, but good luck with developer documentation. There's not so much as Javadocs for their API.

We use Autonomy's IDOL search engine extensively at work and have become sort-of experts on it. Well, to a point. See, Autonomy don't give out help and documentation all willy-nilly. You've got to pester the shit out of them to get anything useful.

Their documentation is shit. There's like, a 500 page PDF of server operations, but good luck with developer documentation. There's not so much as Javadocs for their API.

During this whole process, we've run into capacity issues. IDOL was taking around 16 seconds per index transaction.

I could see an abundance of connections on the Autonomy server in the status of FIN_WAIT2. This means the transaction has completed and it is just waiting for the client (our message processor) to close the connection. We are using Autonomy's API to index data, and the API itself is taking around 16 seconds to complete a connection, while the actual index transaction really only takes a few milliseconds.

I decompiled their code to see that they aren't releasing the connection when they should be which I believe is causing this FIN_WAIT2 problem.

According to the HttpClient documentation:

"Release the Connection: This is a crucial step to keep things flowing. We must tell HttpClient that we are done with the connection and that it can now be reused. Without doing this HttpClient will wait indefinitely for a connection to free up so that it can be reused.
method.releaseConnection();"

Using the exelent Java Decompiler by Emmanuel Dupuy, I was able to see exactly where they weren't calling method.releaseConnection(), and I pointed it out to them. Lawyered.

So now we're just waiting for them to admit they fucked up and give us an updated version of the API.

In the mean time, I talked with our server team to have them decrease the timeout on FIN_WAIT2s from 60 seconds to 3 seconds in /etc/sysctl.conf. I now no longer see the FIN_WAIT2s hanging out there. However, the connections are still taking around 16 seconds to complete, so I still think the API needs to be fixed to release the connection.

I'm wondering if there's a super-hack method to force the connections to close. I don't have access to the HttpClient library, because it is wrapped inside the ACI API, and that is just a JAR, no source. I'm going to do some digging though and see if I can force that to close. Last resort, if they won't fix this for us, is to write my own indexer client. Can't be terribly difficult. I have extensive experience with HttpClient from my previous project. Dare me to do it.

0 comment

Check for New Comments
Slashdot Account

Need an Account?

Forgot your password?

Don't worry, we never post anything without your permission.

Submission Text Formatting Tips

We support a small subset of HTML, namely these tags:

  • b
  • i
  • p
  • br
  • a
  • ol
  • ul
  • li
  • dl
  • dt
  • dd
  • em
  • strong
  • tt
  • blockquote
  • div
  • quote
  • ecode

"ecode" can be used for code snippets, for example:

<ecode>    while(1) { do_something(); } </ecode>
Sign up for Slashdot Newsletters
Create a Slashdot Account

Loading...