Charlie Arehart talks about “The Impact Of Unexpected Load and How To Counter It” in this episode of ColdFusion Alive podcast with host Michael Smith. He was one of the speakers at the CF.Objective Conference. In this session, veteran server troubleshooter Charlie Arehart will guide a more detailed review of the issues above, including how to identify such traffic, more on these specific impacts, and most important identifying the solutions along with their pros and cons. He has helped shops achieve dramatic reductions in impact from such automated requests, resulting in greater server stability and performance.
Are Spiders Eating Your Servers? The Impact Of Their Unexpected Load and How To Counter It
- Where does unexpected load come from?
- How to identify spiders eating your servers
- DOS attacks, script kiddies
- Malformed requests
- What Google Analytics misses in load
- Load balancer testing
- Use a test page rather than probing your homepage every 5 seconds
- What about robots.txt? Doesn’t that block bots?
- SaaS Tools that search for “bad” traffic flooding your server and don’t let it into your server
- Older CFer, older resources
- Why are you proud to use CF?
- WWIT for you to make CF more alive this year?
- What are you looking forward to at CFObjective?
For years I’ve watched people try to tame “server problems” with a focus on their code, their SQL, the jvm, and so on. Yet often it turns out that the root cause is actually unexpected load. And that load may be from things you never expected (automated), at volumes you never expected. I’ve found folks with as much as 80% of their web traffic to be such unexpected automated traffic! Worse, there are characteristics of such automated visits that may actually have MORE IMPACT than “real users”: for instance, did you know they create a new session–and run session startup code–for each page they visit?!
The good news is there are solutions to better manage (or simply block) such automated requests which may already exist in your environment, and tools you may consider (some free, some commercial) which can be easily implemented. There are even SAAS solutions that could help alleviate such problems with just a single tiny change in your environment! You may also want to consider some admin configuration options related to sessions and/or client variables, as well as reconsider some coding choices in your session startup code.
Mentioned in this episode
- Web spiders
- User agent header faking
- IP address spoofing
- On session start
- Load Testing
- Cloud/SAAS Firewall-level Application Firewalls tools list
- CFML Job Resources
- Ortus Solutions
- CF Alive episode on What's New In CF 10, 11, And 2016 That You May Have Missed?
- Mike Brunt
- Scary DBA
- Pass conference
- Database fundamentals
- CF Slack channel
- FB programmers group
- TeraTech blog on CFML online groups
- Mary Jo Sminkey CFObj Advanced Error Handling Strategies ColdFusion Alive podcast episode
- Gert Franz CFObj Debugging ColdFusion Alive podcast episode
- CFML Resource Sites
- Adobe CF blog blogs.coldfusion.com
- 2000 sales of CF per quarter
- CF Summit
- CF Meetup
- CFObjective Twitter hashtag
- Things you can do in DC
Michael: Welcome back to the show. I'm here with Charlie Arehart and he's a veteran ColdFusion troubleshooter. He has been doing I.T. stuff for so long. We don't even want to talk about that until the second half of the show when we will talk about how long we've been doing things and the benefits of that.
But first of all, we're going to talk about unexpected load and how you can counter it and how they may be spiders or other things you didn't even know about. In fact, could you even get to expecting unexpected load? So, lots of crazy stuff there. Charlie’s going to be talking about that at cf.Objective which is coming up soon. So, we'll talk a little bit about that.
So welcome back to the show Charlie.
Charlie: Thank you for having me Michael. Good to see you.
Michael: Yeah, yeah good to see you too coming to us from beautiful downtown Kentucky.
Charlie: Not downtown, beautiful rural out.
Charlie: Yes, you're out in the sticks. You escape from the sick city town that's got fifteen hundred people in it.
Michael: Goodness me!
Charlie: That was from a city that's got you know, fifty thousand people in it. I love it.
Michael: You’re quite a way away from dizziness.
Charlie: And it’s beautiful here.
Michael: You know your land used to have…
Charlie: Right the land and then D.C. grown up in D.C.
Charlie: Forty's and then moving to Atlanta from my forties and now here and working from here. And that's why we came here because with the work that I do, it's all remote. I can connect to people and log you know, as long as I got a decent internet connection. I can work from here and I'm looking out at the beautiful vista of trees and sometimes turkeys and deer and all kinds of fun stuff.
Michael: That’s amazing, it's incredible what technology has allowed people to do and I’m sure some of the listeners are working with remote as well.
A veteran server troubleshooter who’s worked in enterprise IT for more than three decades, Charlie Arehart (@carehart) is a longtime community contributor who as an independent consultant provides short-term, remote, on-demand troubleshooting/tuning assistance for organizations of all sizes and experience levels (carehart.org/consulting).