TeraTech

The ColdFusion Experts: Develop | Secure | Optimize

  • Services
  • About
  • CF Alive
  • Blog
  • Podcast
  • Contact

  • Services
  • About
  • CF Alive
  • Blog
  • Podcast
  • Contact

013 Are Spiders Eating Your Servers? The Impact of Their Unexpected Load and How to Counter It, with Charlie Arehart

April 18, 2017 By Michaela Light 7 Comments

Charlie Arehart talks about “Are spiders eating your servers? The impact of their unexpected load and how to counter it” in this episode of the ColdFusion Alive Podcast, with host Michaela Light.

“These days a lot of people are concerned about site security and they'll … Either they or someone in the organization will run some sort of a scam against their server, but when they run that scan, again what are they doing? They're trying to call lots of requests with different forms of potentially bad break-in type code. And those tools, sometimes they're not as careful and they might make a request every second.” – Charlie Arehart

Contents

  • Episode highlights
  • Mentioned in this episode
  • Speaker details
  • Links
  • Interview transcript

Episode highlights

  • Why your ColdFusion app being slow might not be your SQL, JVM, or even your CF code
  • What exactly is unexpected load
  • What kind of unexpected traffic have you seen
    • Is it a lot of traffic?
    • Is it common to find on CF servers?
  • What can be the crazy problems with unexpected traffic?
  • But, but, what about intranet-only sites – aren’t they safe from unexpected traffic?
  • How you can mitigate these problems fast
  • What about robots.txt? Doesn’t that block bots?
  • Ok, but aren’t spiders and bots getting smarter, so harder to handle?
  • Why are you proud to use CF?
  • WWIT to make CF more alive this year?
  • What are you looking forward to at Into The Box?
  • The move to the countryside

Charlie says:

For years I've watched people try to tame “server problems” with a focus on their code, their SQL, the jvm, and so on. Yet often it turns out that the root cause is actually unexpected load. And that load may be from things you never expected (automated), at volumes you never expected.   I've found folks with as much as 80% of their web traffic to be such unexpected automated traffic! Worse, there are characteristics of such automated visits that may actually have MORE IMPACT than “real users”: for instance, did you know they create a new session/client variables, and run session startup code, for each page they visit?!

The good news is there are solutions to better manage (or simply block) such automated requests which may already exist in your environment, and tools you may consider (some free, some commercial) which can be easily implemented. There are even SAAS solutions that could help alleviate such problems with just a single tiny change in your environment!  You may also want to consider some admin configuration options related to sessions and/or client variables, as well as reconsider some coding choices in your session startup code.

In this session, veteran ColdFusion server troubleshooter Charlie Arehart will guide a more detailed review of the issues above, including how to identify such traffic, more on these specific impacts, and most important identifying the solutions along with their pros and cons. He has helped shops achieve dramatic reductions in impact from such automated requests, resulting in greater server stability and performance.

And to continue learning how to make your ColdFusion apps more modern and alive, I encourage you to download our free ColdFusion Alive Best Practices Checklist.

Because… perhaps you are responsible for a mission-critical or revenue-generating CF application that you don’t trust 100%, where implementing new features is a painful ad-hoc process with slow turnaround even for simple requests.

What if you have no contingency plan for a sudden developer departure or a server outage? Perhaps every time a new freelancer works on your site, something breaks. Or your application availability, security, and reliability are poor.

And if you are depending on ColdFusion for your job, then you can’t afford to let your CF development methods die on the vine.

You’re making a high-stakes bet that everything is going to be OK using the same old app creation ways in that one language — forever.

All it would take is for your fellow CF developer to quit or for your CIO to decide to leave the (falsely) perceived sinking ship of CFML and you could lose everything—your project, your hard-won CF skills, and possibly even your job.

Luckily, there are a number of simple, logical steps you can take now to protect yourself from these obvious risks.

No Brainer ColdFusion Best Practices to Ensure You Thrive No Matter What Happens Next

ColdFusion Alive Best Practices Checklist

ColdFusion Alive Best Practices Checklist

Modern ColdFusion development best practices that reduce stress, inefficiency, project lifecycle costs while simultaneously increasing project velocity and innovation.

√ Easily create a consistent server architecture across development, testing, and production

√ A modern test environment to prevent bugs from spreading

√ Automated continuous integration tools that work well with CF

√ A portable development environment baked into your codebase… for free!

 

Learn about these and many more strategies in our free ColdFusion Alive Best Practices Checklist.

Mentioned in this episode

  • Web spiders
  • Yandex
  • Bidu
  • Yahoo slurp
  • User agent
  • robots.txt
http://traffic.libsyn.com/coldfusionalive/CF_ALive_Podcast_Charlie_Arehart_Twisted_Vawe.mp3

Podcast: Play in new window | Download | Embed

Subscribe: Apple Podcasts | RSS

Speaker details

A veteran server troubleshooter who's worked in enterprise IT for more than three decades, Charlie Arehart (@carehart) is a longtime community contributor who as an independent consultant provides short-term, remote, on-demand troubleshooting/tuning assistance for organizations of all sizes and experience levels (carehart.org/consulting).

Links

  • Website
  • Twitter
  • Facebook
  • LinkedIn

(* WWIT = What Would It Take)

Interview transcript

Michaela Light:               I'm here with Charlie Arehart, veteran ColdFusion server troubleshooter. Hey Charlie. We're going to be looking at some very strange stuff, which is why your ColdFusion might be slow, but it's nothing to do with your SQL JVM or even your CF code. And what we're going to be talking about is an unexpected load.

                                            So, we'll look at that and we'll look at what different kinds of unexpected traffic you may have on your server and how much traffic that could be and how common it is to find. I'll let you into a secret here, it's pretty darn common. And what kind of crazy problems you could have on your server when you have an unexpected load.

                                            You might be thinking, “Well, I have an intranet site, aren't they safe from the unexpected load?” Charlie is going to let out, reveal the secret on that. No, he's shaking his head. So, you better listen to this even if you have intranets. And we're going to look at how you can mitigate these problems fast a lot of the time, other times it might take longer. We'll look at the different kinds.

                                            And you may be thinking, “But what about robots.txt, doesn't that block all these things?” And unfortunately, I think the answer is no again. And also what about all these spiders bots that scan your site, aren't they getting hard as a block and they getting small, so how do you handle a smart spider? It sounds dangerous. So, it's a very interesting topic that you may not have thought about before. So, welcome Charlie.

Charlie Arehart:             [foreign 00:01:36]  

Michaela Light:               [foreign 00:01:39]

Charlie Arehart:             [foreign 00:01:42]

Michaela Light:               Yeah.

Charlie Arehart:             It's going to be lots of Spanish and mariachi and all kinds of fun stuff.

Michaela Light:               Yeah, si.

Charlie Arehart:             Let's not confuse anybody [crosstalk 00:01:55] It's okay if you just speak english.

Michaela Light:               Okay, we'll stay in English. Yeah, we'll speak English. We're just joking around. So, yeah. Usually when people think about server troubleshooting, they usual go to's are, they're looking at their SQL or their JVM or their ColdFusion code, but what you're saying is that … Of all those often our problems on servers, [inaudible 00:02:19] will be something else happening.

Charlie Arehart:             Yeah. And it may be that those things have an impact. It may be that those things are being stressed in ways you don't expect. And that's really the bottom line of this talk. I think the title I had was our spiders eating your server and so that's … What we're getting at is all I do all day, every day is troubleshoot people's servers, whether CF or LUCY or Railo or even some other things.

                                            Most of the time when we're doing things there's some commonality, the configuration issues. Sometimes there's coding issues, it's not as often as people might think I mean. That's another little lessons of this, I would say, is that a lot of people when they have trouble they focus on looking at their code, they focus on looking at their SQL, or if they have gotten past that or ran out of ideas then they start thinking about tuning their JVM. I'm telling you it's really rarely those things.

Read more

 

Related Posts

  • State of the ColdFusion Union 2017 Survey Amazing ResultsState of the ColdFusion Union 2017 Survey Amazing Results
  • Adobe ColdFusion Summit East 2019 AnnouncementAdobe ColdFusion Summit East 2019 Announcement
  • 5 Great Conference Surviving Tips for Introverted Developers5 Great Conference Surviving Tips for Introverted Developers
  • Intro To Couchbase For Caching And NoSQL – Webinar Tues 4/8/2014 1PM EDTIntro To Couchbase For Caching And NoSQL – Webinar Tues 4/8/2014 1PM EDT
  • 074 Planning for CFML ISP disaster (Commandbox and Docker to the rescue) with Mark Drew074 Planning for CFML ISP disaster (Commandbox and Docker to the rescue) with Mark Drew
  • 088 CFWheels (A ColdFusion Framework for the REST of us) with Tom King and David Belanger088 CFWheels (A ColdFusion Framework for the REST of us) with Tom King and David Belanger
  • Facebook
  • Twitter
  • LinkedIn

Filed Under: CF Alive Podcast, IntoTheBox Conference

← Previous Post 013 Are spiders eating your servers? The impact of their unexpected load and how to counter it with Charlie Arehart – Transcript
Next Post → 014 NGINX: A Smart Middle Man Between Your App and Your Users, with Kevin Jones

Subscribe on iTunes

CF Alive Best Practices Checklist

 

Modern ColdFusion development best practices that reduce stress, inefficiency, project lifecycle costs while simultaneously increasing project velocity and innovation.

Recent Posts

  • 4 Reasons Why Your ColdFusion Web Apps Are Suffering (And How To Avoid It)
  • Google Down – An Unprecedented Event (Save Your Data Fast!)
  • 107 ColdFusion 2021 Revealing Details on How it was Created with Rakshith Naresh
  • Into The Box LatAm 2020 Virtual Conference – Free to Register!
  • Slow ColdFusion Applications May Ruin Your Business (3 Steps to Prevent It)

Categories

  • ActionScript
  • Adobe CF Summit
  • Adobe CF Summit East
  • Adobe CF Summit East 2018
  • Adobe ColdFusion 11
  • Adobe ColdFusion 2020 Beta
  • Adobe ColdFusion 2021
  • Adobe ColdFusion Project Stratus
  • Adobe ColdFusion Security
  • AIR
  • Ajax
  • AngularJS
  • Announcement
  • API
  • Apollo
  • Auto Security Lockdown
  • AWS
  • C#
  • Certification
  • CF Alive
  • CF Alive Book
  • CF Alive Podcast
  • CF Camp
  • CF Developer week
  • CF Maintenance
  • CF Summit India
  • CF Tags
  • CF Training
  • CF Vs. Other Languages
  • CFEclipse
  • CFML
  • CFML Open- Source
  • CFObjective
  • cfquery
  • CFSummit
  • CFUnited
  • China Chopper
  • CIO
  • Classes
  • Client Highlights
  • ColdBox
  • ColdFusion
  • ColdFusion 2018
  • ColdFusion 2020
  • ColdFusion 2021
  • ColdFusion 9
  • ColdFusion community
  • ColdFusion Conference
  • ColdFusion Consulting
  • ColdFusion Developer
  • ColdFusion Development
  • ColdFusion Hosting
  • ColdFusion Security
  • ColdFusion Webinar
  • CommandBox
  • Conference
  • Cool Stuff
  • Culture
  • Cybercrime
  • Database
  • Development Approach
  • DevOps
  • Docker
  • Fixinator
  • Flex
  • Frameworks
  • Fusebox
  • FusionReactor
  • Futurology
  • Garbage Collector
  • Google Down
  • Into The Box Latam
  • IntoTheBox Conference
  • Java
  • JavaScript
  • JVM
  • Learn ColdFusion
  • Legacy Code
  • Load Testing
  • Lucee
  • Management
  • MAX
  • MDCFUG Lunch
  • Microsoft Azure
  • Mindmapping
  • MockBox
  • Modernize ColdFusion
  • Monitoring
  • Muracon
  • NCDevCon
  • New Intern
  • News
  • Node.js
  • Open- Source
  • ORM
  • Ortus Developer Week
  • Ortus Roadshow
  • Performance
  • Performance Tuning
  • PHP
  • Productivity
  • Programming Languages
  • Project planning
  • Query of Queries
  • Roadmap
  • Scalability
  • Security
  • Server Software
  • Server Tuning
  • Social Media
  • Spiral Web
  • SQL
  • Success Story
  • Survey
  • Technology
  • TestBox
  • Tips
  • Transcript
  • Trapeze Development
  • Uncategorized
  • Web 2.0
  • Web Application
  • Web Server
  • Webinar
  • Webmail
  • What is ColdFusion?
  • Whole Brain Development
  • Women in Tech
  • Work From Home

Recent Comments

  • Michaela Light on A Comprehensive Guide to Running a Successful CFML Project
  • Michaela Light on Is Lucee CFML now better than Adobe ColdFusion?
  • Michaela Light on Introducing Swansea Jack (Lucee CFML 6 announced)
  • Michaela Light on 082 ColdFusion and the Blockchain Revolution with Mike Brunt
  • Michaela Light on 082 ColdFusion and the Blockchain Revolution with Mike Brunt
  • Home
  • Services
  • About Us
  • CF Alive
    • CF Alive Book
    • CF Alive Inner Circle
    • CF Alive full resources cheatsheet
  • Blog
  • Podcast
    • Podcast Guest schedule
  • Contact
  • Sitemap

The ColdFusion Experts:
Develop, Secure, Optimize

TeraTech Inc
451 Hungerford Drive Suite 119
Rockville, MD 20850

Tel : +1 (301) 424 3903
Fax: +1 (301) 762 8185

Follow us on Facebook Follow us on LinkedIn Follow us on Twitter Follow us on Pinterest Follow us on YouTube

Copyright © 1998–2021 TeraTech Inc. All rights Reserved.