Top 10 Architecture Scalability Mistakes Made in ColdFusion
ColdFusion, Software, MySQLThis is a focus on architecture design mistakes I've seen made too often over the years by CF developers building ColdFusion architectures. Often times, the codebase and system started small and grew too fast, such that the original developer/architect didn't have the time or experience to scale the system up with the demands of high-load applications.
- Not dedicating enough time and resources on optimizing the database:
Too often, there are smart, but unqualified programmers/developers designing and maintaining the database. A qualified/certified DBA, even part-time, is well worth the investment. You need one who can profile your database, tune all the server and individual database settings, recomming index changes, be responsible for DB design and integrity, regular maintenance, set up replication/failover/clustering, and will proactively monitor your database to let you know what problems need attention in relationship to your applications. - Not dedicating enough time and resources on optimizing queries:
Again, developers tend to throw themselves into the role of writing all the queries, but don't necessarily think about the performance/scalability issues of writing non-optimal queries. Often times a DBA who is qualified for all the Database Server stuff (see #1 above) isn't the same type of DBA who can optimize your queries. You need someone who can run the queries with explain plans, and then can make recommendations on improvements. These improvements pay off immensely! - Coupling all parts of the system into one mega-application, instead of separate applications
Usually they start out as fine: clever little applications that "do the job". They may have started in the CF 4 or 5 days, and grow and grow over the years until they're a mass of spaghetti code. It may never occur to the developers that as the app gets bigger that not everything belongs together any more. It will take some work, but it's important for long-term stability and scalability that you de-couple big chunks into smaller applications that run independent from the original beast. Scheduled Tasks, Web Services, APIs, User Admin Areas, Reporting, file/image services, Search services, etc. should no longer be coupled to the main application. Each of those parts should be broken out, so that each can be tuned optimally. Do you want your whole system to start bottlenecking because Search or Web Services become slow or go down? If they're decoupled, you don't have that problem. - Not using a good "shared" file system scalable for high numbers of file writes
When there are multiple physical servers in your cluster (if applicable), you may want a single location to save files. No-no's here are mapped drives/mounts to a non-dedicated server (such as a drive on your database server, some external drive, etc.). A good NAS and/or SAN solution with failover scales better as file writes become higher over time. Cloud Computing solutions are becoming big now, too, but I haven't tested enough in this area to make sound recommendations. I'm sure other with more experience can tell us. Comments? - Not tuning the JVM
Many articles have already been written on this subject, so I won't go into it. If you're having performance problems and you don't know how to tune your JVM, get help from one of the many articles out there, hire an expert, or get Adobe Platinum support for CF. - Not tuning Request settings in the ColdFusion Administrator
See #5 above. - Not making good use of Multiple Instances
When you de-couple areas of your application (see #3), often it's wise to put your new applications into separate CF instances. For instance, all heavy tasks (Scheduled Tasks and other "batch" type processes) should run on a separate instance. You don't want to know how many times I've seen a Scheduled Task bring the entire infrastructure to it's knees. - Not using CF Monitoring tools, such as SeeFusion, CF8 Server Monitor, and/or FusiontReactor
Get familiar with at least 2 of these, and hire an expert to help you get up to speed on them more quickly. These save your butt! - Trying to recreate the "holy grail" reporting system from scratch
First, any reporting system should be a prime candidate for de-coupling. And reports that come from the database should not be running against your "write" (master) database, but rather against your "read-only" (slave) database, or else you'll have a lot of bottlenecks on your hands as users create massive reports reading data while other users are trying to simultaneously write data via the rest of the application.
But secondly, look for reporting solutions that are already available. Most of the time users don't really need *real-time* reports, and off-line/archived data lends itself better to the more complex reports that users start to request over time (allowing Excel exports makes a lot of people happy, cuz they know how to generate reports "the way they like them" in familiar Excel. Also, don't write a Report UI that allows users to request *too many* records - your milage will vary, but it's almost never a good idea to allow reports that have no filters, such that they can just *get everything* all at once. Few web apps are built for handling that much data, so find other solutions. - Not taking advantage of various caching capabilities:
We all know (hopefully) to cache queries where possible, but... do you use the Application/Server scope where appropriate, and to use cfcache (or other 3rd party CF cache tags/CFCs/frameworks) in areas that rarely change? Do you come up with caching strategies for data, so that it can persist in the Application/Server scope without going back to the dB unnecessarily? Could other data/files be cached to disk? Can you use Verity (or Lucene/Solr) for searches instead of pulling directly from the database all the time?
Add Comment
OPTIMIZE all MySQL database tables via ColdFusion
ColdFusion, MySQLEver want to run a script that does an OPTIMIZE of all tables in your database? First off, you can do this with a shell script or cron job with something like: "
mysqlcheck -u root -p --auto-repair --check --optimize --all-databasesBut what if you, say, don't have a full-time DBA on staff, and want to run these on a schedule which you can easily control with ColdFusion? That was our situation, and so here's what I came up with. We ran it last weekend during our scheduled "maintenance window" and it took about 90 minutes with no downtime on a MySQL 5.1 18G database
(disclaimer: the speed of this process may vary, depending on numerous variables such as hardware, memory, version of MySQL, etc.)
<!--- set these according to your database size: 14400 and 1800 were used after extensive testing
on a system with 8G ram and quad core in 2009 for an 18G database; you can easily try different
values here by trying them in the URL
(ex: /OPTIMIZE_ALL_DB_tables.cfm?queryTimeout=5&numOfTables=5) --->
<cfparam name="URL.RequestTimeout" default="14400">
<cfparam name="URL.queryTimeout" default="1800">
<cfparam name="URL.myDatasource" default="my_database">
<cfparam name="URL.numOfTables" default="1000000">
<!--- set u/p to one that has permissions to run OPTIMIZE & FLUSH on your tables --->
<cfset Variables.myUsername = "db_user">
<cfset Variables.myPass = "my_password">
<cfset Variables.toAddresses = "webmaster@mydomain.com;aaron@mydomain.com">
<cfset Variables.fromAddress = "webmaster@mydomain.com">
<cfsetting requesttimeout="#URL.RequestTimeout#">
<!--- get host and instance if they don't already exist --->
<cfif NOT IsDefined("Server.hostName")>
<cftry>
<cfset local.oInetAddress = CreateObject("java", "java.net.InetAddress")>
<cfset Server.hostName = local.oInetAddress.getLocalHost().getHostName()>
<cfcatch type="application">
<cfset Server.hostName = "Could not determine with java.net.InetAddress">
</cfcatch>
</cftry>
</cfif>
<cfif NOT IsDefined("Server.instanceName")>
<cftry>
<cfset local.oJRun = CreateObject("Java", "jrunx.kernel.JRun")>
<cfset Server.instanceName = local.oJRun.getServerName()>
<cfcatch type="application">
<cfset Server.instanceName = "Could not determine with jrunx.kernel.JRun">
</cfcatch>
</cftry>
</cfif>
<cfquery name="rsAllTables"
datasource="#URL.myDatasource#"
username="#Variables.myUsername#"
password="#Variables.myPass#">
SHOW TABLES IN peoplexs_test;
</cfquery>
<!--- <cfdump var="#rsAllTables#" format="text"> --->
<cfset processInfo = "">
<cfoutput query="rsAllTables" maxrows="#URL.numOfTables#">
<cfset dbErrorInfo = "">
<cfset anyErrorInfo = "">
<cfset beginProcessInfo = "">
<cfset endProcessInfo = "">
<cfsavecontent variable="beginProcessInfo">
--- <strong>"OPTIMIZE TABLE #rsAllTables.TABLES_IN_PEOPLEXS_TEST#" started</strong>... <br/>
</cfsavecontent>
<cfset processInfo &= beginProcessInfo>
#beginProcessInfo#
<cfflush>
<cftry>
<cfquery name="qOptimizeTable"
datasource="peoplexs_test"
username="#Variables.myUsername#"
password="#Variables.myPass#"
timeout="#URL.queryTimeout#">
OPTIMIZE TABLE #rsAllTables.TABLES_IN_PEOPLEXS_TEST#;
</cfquery>
<cfquery name="qFlushTable"
datasource="peoplexs_test"
username="#Variables.myUsername#"
password="#Variables.myPass#">
FLUSH TABLE #rsAllTables.TABLES_IN_PEOPLEXS_TEST#;
</cfquery>
<cfcatch type="database">
<cfsavecontent variable="dbErrorInfo">
<span style="color:red;">
Database ERROR: "#cfcatch.Cause.Message#" on
"#cfcatch.SQL#"
</span><br/>
</cfsavecontent>
<cfset processInfo &= dbErrorInfo>
#dbErrorInfo#
</cfcatch>
<cfcatch type="any">
<cfsavecontent variable="anyErrorInfo">
<span style="color:red;">
<cfdump var="#cfcatch#" format="text">
</span><br/>
</cfsavecontent>
<cfset processInfo &= anyErrorInfo>
#anyErrorInfo#
</cfcatch>
</cftry>
<cfsavecontent variable="endProcessInfo">
------ done with #rsAllTables.TABLES_IN_PEOPLEXS_TEST#. <br/><br/>
</cfsavecontent>
<cfset processInfo &= endProcessInfo>
#endProcessInfo#
<cfflush>
</cfoutput>
<cfmail to="#Variables.toAddresses#"
from="#Variables.fromAddress#"
subject="OPTIMIZE_ALL_DB_tables.cfm on #Server.hostName#/#Server.instanceName# at #DateFormat(Now(), 'yyyy-mm-dd')#"
type="HTML"
server="localhost">
<cfoutput>#Trim(processInfo)#</cfoutput>
</cfmail> Add Comment
MultiCFAdmin Alpha 0.007 is on RIAForge.org now
ColdFusion, Software, MultiCFAdminHere's the link: http://multicfadmin.riaforge.org/
Description:
In this Alpha version 0.007, which allows connection to CF Admin API on local and remote CF instances, there are only a few features:
1) jQuery UI
2) ability to see basic stats of each CF8 Server Monitor API
3) ability to clear Template Cache for all instances
4) ability to toggle Trusted Cache for all instances
5) can configure multiple CF Admin URLs, passwords, and a sleep() time for how long MultiCFAdmin waits after Trusted Cache is changed
Requirements:
* multiple CF instances other than cfusion (default)
* access to jrun-web.xml configuration files on each instance
Add Comment
Multi-CF Admin Alpha 0.02
ColdFusion, Software, MultiCFAdminOkay, this is pretty cool (at least I think so). Here's the zip.
How many of us with multiple instances in a cluster have wanted to be able to do things like clear the template cache for every instance in the cluster with *one* button?
Well, I have.
Now, I have an early alpha that is not much more than a proof-of-concept, but does exactly that.
and in a few seconds, it's all cleared (see below)...
Here's the readme.txt:
Requirements:
* ColdFusion 8.01 Enterprise or Developer, Multi-server
* multiple CF instances other than cfusion (default)
Installation:
1.) put /MultiCFAdmin folder at www root
2.) open jrun-web.xml file at [jrun_root]\servers\[instance_name]\cfusion.ear\cfusion.war\WEB-INF (you may need to create it by copying over from cfusion instance if it doesn't exist)
3.) add a line within the
���
���
4.) NOTE: don't attempt to connect to the cfusion instance because it causes errors with the Web Service calls to try and connect from the same instance
5.) set environment info for Application.Environment in config/Environment.cfm
6.) set instance info in config/CFInstances.cfm
Here's the zip again.
Send me a tweet if you're interested in helping with the code: http://twitter.com/aqlong
Verity "java.io.IOException: Read timed out" error?
ColdFusion, VerityHi anyone,
On CF 8.01 (CHF 2), Enterprise 64-bit, multi-server on Red Hat 5.3 I'm getting the following error whenever I try to do any action with a Verity collection:
An error occurred while performing an operation in the Search Engine library.
Error switching collection offline.: com.verity.api.administration.ConfigurationException: java.io.IOException: Read timed out
- By "anything", I mean creating, optimizing, deleteing, reindexing, etc. from the CF Admin or regular CF code.
- I've restarted CF and Verity multiple times to make sure it wasn't that
- I tried on 2 different machines, each with the same exact setup and config, and it happens on both, so I don't think it's specific to a machine
- I tried using the /tmp directory, which has pretty much full rights, as the path for creating some brand new indexes, but still get the error
- The strange thing is that this wasn't happening about 4 weeks ago, so I'm thinking this is probably an O/S (RHEL 5.3, 64-bit) update issue, or perhaps occurred when I applied CHF 2 for CF 8.01 a few weeks ago, though there's nothing about Verity in it.
Any ideas what it could be or what to try next? I'll be calling Adobe support shortly, but wanted to see if any of my blog followers had an idea?
TIA
I'm on twitter now: aqlong
ColdFusion, SoftwareIf you care to follow me on twitter now, my account is aqlong
Loading....