Load List Values for Improved Efficiency 207
An anonymous reader writes "Reduce the number of database hits and improve your Web application's efficiency when you load common shared list values only once. In this code-filled article, learn to load the values for drop-down lists when your Web application starts and then to share these loaded list values among all the users of your application."
You know what's great (Score:3, Insightful)
Re:You know what's great (Score:2)
In persistent perl:
{ #note extra scope
my @dropdowndata;
sub whatever{
#call sub/method unless you already have the cached version.
@dropdowndata ||= load_dropdown_data();
}
}
Re:You know what's great (Score:2, Funny)
Re:You know what's great (Score:2)
Re:You know what's great (Score:2)
... absolutely not. A database is not the end-all and be-all of computing, not when static or semi-static information, or information that is easily maintained as text/html.
I recently had the "opportunity" to test out a system that stored live video in a database - majorly dumb idea. We ate the money we spent on it - hardware and software - but if I had known ahead of time that's what they were doing, it w
Re:You know what's great (Score:4, Insightful)
If you take your example of codes in a file, how will you distribute it? How will you maintain it? How can multiple users access it? What happens if someone accidentally deletes it?
Databases are designed to solve exactly these types of problems.
You need to have the One True Copy of the data, in all cases. If you wish to distribute a flat-file or marked up copy of that data provided it's completely static aside from a software revision, then that's fine. But keeping key data unprotected -- or worse, opening up yourself to multiple masters -- demonstrates to me that you're thinking about things at the hobby/toy project level, certainly not at a distributed or COTS level.
Your data is your application. It doesn't matter if the data is static or not. And for goodness sakes, if you're going to be using a database anyway for the dynamic data, eat the storage and keep the master copy in the freaking database!
(And if you're not using a database at all, well, that's further evidence to me that you're thinking at the toy/hobby level of project scope.)
If you were going to store large, binary data sets (such as video) in a database, I'd tell you to use Oracle and either store the data out-of-line with BFILEs, using the database as an indexing mechanism, or store it in the data as BLOBS, depending on whether access time (use BFILES!) or recoverability (use BLOBS!) was more important. I don't know what that project used, but writing either of the apps with those data storage concepts is pretty trivial.
Re:You know what's great (Score:2)
Changes to the lists? (Score:5, Insightful)
Re:Changes to the lists? (Score:3, Informative)
Not to say that the article was actually bad or anything, its just a little light on when you would want to use this, and what some of the problems with this approach are.
Re:Changes to the lists? (Score:1, Insightful)
Really? What about articles on the web or posts here in Slashdot? They almost never change and are usually stored in the database.
Re:Changes to the lists? (Score:1)
It doesn't matter that the content of the articles doesn't change, the whole "list of articles" keeps growing as new articles get posted, hence the need for a database.
What this article is about is data that doesn't change very ofter (or at all). I
Re:Changes to the lists? (Score:4, Insightful)
Apples and oranges.
You're talking about adding items (posts) to a recordset (slashdot thread). The items are static but the recordset is not. It changes frequently.
The caching they're talking about involves a recordset that seldom changes, and would therefore be suited to storage outside a database and rebuilt as it changes - IE one trip to the database per change rather than one trip per view. This wouldn't make sense with something like a slashdot thread where records are added non-stop...
Re:Changes to the lists? (Score:2, Informative)
Understand the nature of your data.
In any system, there's basically three kinds of data:
-- Static: This is the stuff that changes at a glacial pace, such as state codes, currency codes, and so forth. (for bonus points, put all the static code/description values in a single table with a type identifier for an even larger performance increase at the cost of slightly more complicated code.)
-- Configuration: This is the data that drives t
Re:Changes to the lists? (Score:5, Insightful)
My data does change, but I store it in a tree.
When someone changes a dropdown, I just erase the cache. When someone wants the list, it gets refilled. All of it is safely synchronized.
Of course, I don't believe it's worth an article, and I don't believe it belongs in
IBM developerworks is a nice source of information when you want to program the mainstream way. It's good for teams, because it makes easily understandable code. Of course it's boooring and not news, though.
Re:Changes to the lists? (Score:2)
One feature I love about ColdFusion is that you can cache queries with a time limit. So if you have a busy app, you can cache the query for even 60 seconds and it'll make a difference.
And yes, I know you CAN do that in PHP and the same with the ADODB library too. (One lib I don't think any php d
Data integrity (Score:5, Insightful)
For instance, the list of US states and Canadian provinces does not change very frequently. I think the last Canadian change was about a decade ago, the last US change nearly 50 years ago.
The "full name, abbreviation" name used in most pulldown lists (full name as label, abbreviation as value) can obviously be considered static.
So why keep it in a relational database? Simple - you can use it to provide referential integrity for all "state" fields in the rest of the database.
This isn't a huge deal with states, but it can be very important with domain level enumerations. Your form actions may be well-behaved, but a robust system must account for clowns who feed their own data directly into your action URL.
(As an aside, this isn't a theoretical problem. I've heard stories of people getting an order form for, oh, a laser printer. They capture it, change the price of the printer from $499.99 to $49.99 and submit the order. The action accepted it, and when the company attempted to refute it they lost because it was considered a bona fide negotiation since the web site could/should have been programmed to reject forms with altered prices. They made an offer, the client made a counteroffer, and the company accepted it. This depends on your state, etc. Given the current political climate I wouldn't be surprised to learn that this is now considered computer fraud with a 10 year prison sentence.)
Re:Data integrity (Score:2)
Having created an online store application, I have struggled with security issues. If it weren't for those, it would have easily taken 1/2 the time that it did to create the application.
One of the main things I concentrated on for the back-end db piece was to verify the data that comes in against the DB itself, and it never accepts data from the end user on pricing (whether or not it's in the form or hidden in the form).
I built the application with two assumptio
Re:Changes to the lists? (Score:2, Interesting)
Basically, create a session variable for the user (we'll call it cacheTimeStamp) with the current date/time to the hour (i.e. yymmddhh).
When the page with the drop-down list is called, check current timestamp. If it's expired, create a new javascript
Then, when the <script> tag is called, use thecacheTimeStamp value to ensure a current
Re:Changes to the lists? (Score:2)
Granted, you could also write it in Java, just saying that its readily available in
Re:Changes to the lists? (Score:2)
Re:Changes to the lists? (Score:2)
I would imagine the benefits of caching would be outweighed by the drawbacks of maintaining a custom caching solution in most cases unless profiling the application has shown that it is absolutely necessary.
Re:Changes to the lists? (Score:2)
end
before_save
def invalidate_root_cache
@@root_cache = nil
end
end
Being that that was a lot of java code, this is an example in rails. before_save marks a method that should be framework-invoked before commiting a change to the database on the categories table (which is introspected from the 'Category' class name, but can of course be overridden). This caches the category list and expires it whenever it may have changed.
This can of cour
Re:Changes to the lists? (Score:2)
Re:Changes to the lists? (Score:3, Informative)
While working on high performance web apps where we want to cache the data and prevent having it become stale we genereally to store it in the application context ( ServletContex
Huh? (Score:5, Insightful)
What's the point? Since when is Slashdot a forum for random tech tips (and not very thrilling ones at that)? Did IBM pay to get this posted? Is Slashdot trying to make fun of IBM by actually posting it?
Re:Huh? (Score:4, Funny)
It all depends on how tight your tin foil hat is.
Re:Huh? (Score:5, Insightful)
I think you're being FAR too polite here, sir. Feel free to drop in an occasional, "Are you f-ing kidding me with this drivel?" in your critique of this type of ridiculously simplistic and obvious article.
On the other hand, there's a good take-away here. If this "revolutionary technique" was so mind-bending to IBM consulting services, I know where I won't be spending my consulting dollars...
Re:Huh? (Score:2)
It's interesting though, that this kind of "revolutionary technique" can be indeed a step forward in the I-don't-know-how-many projects where developers aren't aware of these simple issues. I'm in the consulting business, I've seen them. For them, this kind of consultant might actually be worth the money, and ther
Re:Huh? (Score:2)
If it went into a discussion on how to store your data in memory so that the data the application is looking for is probably in the processor cache, we can definitely talk best and brightest.
Simple caching, however, is basic stuf
Re:Huh? (Score:2)
Absolutely agreed. And the things that you mention are those that you'd wish your expensive IBM consultant was capable of. I stand by my statement however, that
Re:Huh? (Score:2)
The observation that doing things wrong doesn't always run one's employer into the ground does not magically justify doing things wrong. It's this acceptence of adequecy that's behind the poor state of many things today, but that's a rant for lat
Re:Huh? (Score:2)
In the real world of low pay and low prestige IT work it is fairly common for things to be done in a far less than optimal manner. People who know how to do better may just not bother, why make it complicated if it meets the spec - it's not like anyone is gonna give you a cookie if you build an order entry system that is twice the speed they expected.
At the other end programmers just out of school or training programs may have learned some theory and muc
Re:Huh? (Score:2)
Larry Wall was right - good programmers have a sense of Hubris, and want to make their programs "right" whether anyone else will notice or not.
What, The Invention of Cache? (Score:2, Funny)
Re:Huh? (Score:2)
Not me, sir. I learned my trade on totally different languages; I was just referring to Java hashtables because the article was about Java to begin with. For languages such as C, I'll happily grant you another order of magnitude if you wish :-)
Oblig. Simpsons Quote (Score:5, Funny)
"Garbage in the garbage can. Hmm. Makes sense."
Mod parent up! (Score:2)
Re:Oblig. Simpsons Quote (Score:2)
Obligatory paraphrased Dilbert quote (Score:2)
The son of the PHB: My dad always told me to eat the last pickle before drinking the pickle juice.
Dilbert: Something something something!
The son of the PHB: Do you have any idea how much it hurts if you get a pickle in your eye?
Hahahahah! Well, I guess you had to be there.
Point is, the article should have been in developers, not the main page.
Slow news day? Christ. (Score:5, Insightful)
April Fools... (Score:5, Funny)
Wow, next an article about using "for" loops? The benefits of "bubble sort"? "Binary trees"?
Re:April Fools... (Score:2, Funny)
Someone might hold the patents for those techniques.
Well duh! (Score:3, Insightful)
Seriously, this is probably good advice for someone just starting out programming, but I would expect anyone with any experience at all to know about this. Its hardly a revolutionary new technique.
Re:Well duh! (Score:3, Insightful)
Done this for years (Score:5, Insightful)
In ASP.NET you can even do cache invalidation when the database changes. Simply create an extended stored procedure that's fired when any of you update/insert producers run that write to the changed record ids to a Queue (using Microsoft's Messaging and Queuing service) then have a thread in the ASP.NET process that periodically check the queue for new messages and clear the values that have changed out of the cache.
Because the Queuing service works across networks it's a really neat way to provide scalabity in web applications - if you can't wait for SQL 2005 which will provide cache invalidation on database updates as standard.
Simon.
Re:Done this for years (Score:2)
public void doGet(HttpServletRequest req, HttpServletResponse resp) {
ServletContext ctx = getServletContext();
ctx.setAttribute("dataKey", dataObject);
Object dataObj = ctx.getAttribute("dataKey");
}
Funny, I never thought I would compliment an
PHP-ADODB Caches these queries (Score:4, Informative)
[first useful post?]
Re:PHP-ADODB Caches these queries (Score:2)
You answered my question b4 I asked, partially .. (Score:2)
So, you can save result in a file, but can yo usave them in memory too, using PHP?
Re: (Score:2)
Re:You answered my question b4 I asked, partially (Score:2)
Re:You answered my question b4 I asked, partially (Score:2)
It seems that php docs state:
" Optionally you can use shared memory allocation (mm), developed by Ralf S. Engelschall, for session storage. You have to download mm and install it. This option is not available for Windows platforms. Note that the session storage module for mm does not guarantee that concurrent accesses to the same session are properly locked. It might be more appropriate to use a shared memory based filesystem (such as tmpfs on Solaris/L
Re:You answered my question b4 I asked, partially (Score:2)
This should be read: "Do not use." Unsafe concurrent access is a great way to introduce critical bugs that are only found after the app is deployed to production servers. (Just ask any ColdFusion developer who has used sessions in CF 5.0 or earlier.)
Java Servlet init question .. (Score:2)
ListValuesLoader listValues = new ListValuesLoader();
but no global.
However, I can't find the object where the data is actually cached. Is it a Singleton somewhere? Pretty well hidden it must be
Re:Java Servlet init question .. (Score:3, Informative)
That is, they have static 'getInstance' methods, maintain private static references to their own solitary instance, and have private constructors.
This is a very standard OO pattern, and seems to be one thing the article got essentially right.
SQL (Score:2)
But I may be the Kurtz [wikipedia.org] of SQL...
Is This Consultant on YOUR Payroll? (Score:5, Informative)
If you have yet to read the 25-page FA, may I present the precis:
The simplicity of the point however is lost in the complexity of the article. It covers web.xml settings, servlet classes, list value loaders, persistence backends for said loaders, data source 'helpers' for said loaders, custom object classes for the loaders, several subclasses for said object classes, and a jsp page (to boot). Phew.
The author refers to this design as a quick and easy approach. It is not. If you are not familiar with Java and read this article, please do not be put off. He could have demonstrated the point with a far simpler example. E.g. static variable, sql statement, jsp code, done.
[The author] has worked with IBM Global Services for one year, and has five years of experience in J2EE-related technologies. And it shows. I dread to think how much a fully realized IBM Global Services project would cost should all its consultants apply this sized sledgehammer to each small task. Hopefully the article was not written up on the client's dime as well.
--
Java Hosting [rimuhosting.com]
Re:Is This Consultant on YOUR Payroll? (Score:2)
Re:Is This Consultant on YOUR Payroll? (Score:3, Funny)
Re:Is This Consultant on YOUR Payroll? (Score:2)
And, to realise that those consultancies charge a large amount of cash because their developers are crap and produce this over-engineered rubbish. (and is another reason why all those large-scale government projects th
Re:Is This Consultant on YOUR Payroll? (Score:2)
P.S. Before you mention synchonization remeber as of Java 1.2, to get a synchronized Map (in place of Hashtable) or List call Collections.synchonizedMap(map) and Collections.synchronizedList(list).
GoF Decorator pattern is better for this task (Score:5, Insightful)
A Data Access Object should have 1) an interface (defining add, remove, retrieve, etc.) and 2) a standard implementation of the interface that reads/writes to the database on every method invocation.
A Decorator can implement the data access interface, delegating all method invocations to a wrapped instance of the standard implementation. Decorate the behavior of the standard impl. by providing a cache, checking the cache before retrieving a model and updating the cache before saving a model.
Because the standard impl. and the decorator share the same interface, you can have a factory create instances for you. Your code doesn't know or care which instance it is using. Mix and match Decorators to your heart's delight. A logging Decorator (track what data is being access, etc.) can be thrown into the mix, and again your calling code wouldn't be the wiser.
This pattern is easily unit tested and load tested. It doesn't require a running web container to test or run. It Just Works(tm).
Re:GoF Decorator pattern is better for this task (Score:2)
If you think it's overkill then I suggest you haven't understood. It's still just some cacheing code, but its put in a decorator class so it is useful to all callers. It also has the benefit of being easy to prevent dirty reads because it knows about writes and updates.
Pretty damn cool idea if you ask me. (and one which I will be playing with myself when I get back from work!)
thanks to the grandparent poster for mentioning this.
Well, duh (Score:5, Insightful)
I've not RTFA, so perhaps it's truly excellent, but why the hell has this been posted? Anyone who's writing any sort of application and not making intelligent use of caching is either really junior, or should probably be looking for a new job.
Re:Well, duh (Score:2)
Re:Well, duh (Score:2)
Firstly, a lot of your caching solution should depend on what tools or language features are available for the language you're using to implement your site. For example, advocating caching things to disk isn't really necessary when you're using Java since it's more effective to use in-memory caches. If you use an ORM solution like Hibernate, caching happens behind the scenes and you don't even have to think about it. However, if you're using a script
Re:Well, duh (Score:2)
Serializing data to the disk is a good suggestion in many cases. Perl is fairly rich in this area, however, PH
Re:Well, duh (Score:2)
Why would you bother with delimiters when PHP is perfectly capable of serializing arrays/hashes. Just dump all your variables into an array and serialize. Th
Memcached (Score:3, Informative)
Re:Memcached (Score:2)
News? (Score:3, Funny)
By same token, if you're on of those twits who caches five years of data warehouse information in the application layer, there's a special place in programmer hell for you...
Not that bad a hint. (Score:2, Interesting)
Re:Not that bad a hint. (Score:4, Funny)
Looks like you had an SQL Statement timeout there.
And for the follow-up article... (Score:5, Funny)
And what's up with the "obj_" prefixes in some of the code listings? Newsflash: Java is an object-oriented programming language, and most competent Java programmers can figure out that a variable called resultSet probably isn't a primitive. You don't need to waste another 4 characters pointing out that fact.
Underscores for instance variables (Score:2)
Re:Underscores for instance variables (Score:2)
private int _something;
public void setSomething(int something) {
_something = something;
}
vs.
private int something;
public void setSomething(int something) {
this.something = something;
}
Also, it keeps you from having to track down an annoying type bug...
private int something;
public void setSomething(int soomething) {
this.something = something;
}
In
Re:Underscores for instance variables (Score:2)
Personally, I'm not a big fan of glyphs on variable names; though this brings to mind Ruby's use of @ and @@, and isn't all that offensive.
In my own code, I never reuse the same name for local and instance variables.
recursion, indirection, and... (Score:2)
of course, i immediately made a base case, trapped the fowl and filled my belly (to put into practice such sound advice, you see ;-).
mmmm, cs stew...
What is this post for? (Score:2)
Performance? (Score:2)
Though I kid, I find that populating drop-down boxes for applications are pretty minor in the grand scheme of things. My performance issues tend to stem around doing cross-server database queries involving tables with millions of rows. That's when things get slow.
I'm sure there are some people who have to do this level of optimization - folks like Google and Amazon.
What's your Vector, Victor? (Score:3, Insightful)
Bad, bad, bad.
What's with the Vectors, anyway? I haven't used those in years.
It's a lot worse than just the Vectors (Score:3, Informative)
Breakthrough (Score:2)
Externalize Picklists... (Score:3, Insightful)
If the picklists are at all updateable while the application is running, he can cache as he does, but he'll a mechanism to invalidate the cache and re-read from the database.
Forgetting that for a moment:
and
(quotes from http://www.vanderburg.org/Misc/Quotes/soft-quotes
Re:Externalize Picklists... (Score:2)
Why the hell not? Is it uncool now? Do I need to stop using it for l33t status?
There are some nice tag libraries, so STOP PUTTING JAVA CODE IN JSPS!
One might argue it is easier to read especially for something simple.
Are you one of those types that spend all thier time critizing other people's code just because it is not how they would code it? I hope I never work with you.
Re:Externalize Picklists... (Score:3, Informative)
The classic example is:
vector.size();
vector.get(0);
Each one of those calls is synchronized internally but the JVM can still switch threads inbetween the two calls causing a race condition. To make that code thead safe you need to synchronize externally:
synchronized(vector) {
vector.size();
vector.get(0);
}
Re:Externalize Picklists... (Score:2)
Re:Externalize Picklists... (Score:3, Insightful)
Yes. It's uncool. If you wish to be 1337, you need to stop using Vector. Only \/\/4|\|k3rz use Vector. At least that's what they told me in my Java class at DeVry(1). ** braces for DeVry graduate flames **
Vector has setSize, ArrayList doesn't (Score:2)
An article about caching on a geek board? (Score:2)
Come on guys, you can do better than this!
um, right... (Score:2)
The point of stuff like this is, you can keep a dynamic page to generate the html and just make the damn thing an include. You can always r
This time, shoot the messenger (Score:2)
So please direct all flames (A) to whoever posted this as "News", and (B) to whoever accepted this "story".
Or perhaps Hemos said to himself "Damn, a lot of sites sure are slow latley. Perhaps they know not of caching." and then posted the story.
Database hits (Score:2)
Linux uses all otherwise unused memory for file caching. My database machines are loaded with 3-4GB of RAM each, so there is lots of memory used for caching.
PostgreSQL uses shared memory caches to increase performance (though I don't know the exact nature of its shared memory usage), and it work
Not as unnecessary as you think (Score:2)
* feel free to substitute VB here, as I've worked with young programmers who only learned VB in college for whom this approach is beyond their comprehension. Their normal solution is "More RAM".
Its just optimization (Score:2)
Rule 2: Never do anything twice. (cacheing)
Rule 3: Empty cache that won't be required. (space optimization)
If you design properly, your software will run fast, stay fast and small.
Remember "Dr. Dobb's Journal of Computer Calisthenics and Orthodontia (Running Light Without Overbite"?
Its just as true now as it was back when you were writing for machines with K (not M) of RAM.
with Smarty... (Score:2)
You can even use Smarty's html_options function to build your lists for you.
And when something changes the data for the list, just clear the cache. The next time it's needed the cache will be updated.
Seems like a bit of trouble to something... (Score:2)
For instance, on some sites that I've developed, whenever an admin page is used to add to menus, it *does* update the DB, but it also recreates a static
Or did I miss the point of the article?
Is this a real problem? (Score:2)
The more likely situation is that the Java fanatics got carried awa
Re:Almost ...can be viewed as an exercise in cachi (Score:2)
A smart programmer can do very good compression. A wise programmer will do it so that you can add likely stuff later and recompress without taking too much time (perhaps at the expense of lower compression).
Of course if it's bad programming it's an exercise in data loss, corruption or explosion
Re:seems pretty self-evident (Score:2)