You can listen to the podcast and read the show notes here
Michael: Welcome back to the show. I'm here with Brian Klass, and he is speaking at C.F objective in a few days' time about how you can level up your web apps with Amazon Web Services. And before you yawn, there’s so many things in Amazon Web Services you probably didn't even know are there. And we will talk a bit about that and all the amazing things you can do with S-3 storage. And he is going to be demoing at C.F. objective how you can use ColdFusion and 00:28 [Node J S micro-services to invoke on demand way so, you can use a WS lambda, and we'll talk about what that is.
And also, we've got some artificial intelligence stuff you can call from your app as well which is processing images using a thing called ‘reflector’. And we'll also look at some notes sequel data store using dynamo D.B. And he's going to be talking about common problems you can avoid when you're dealing with cloud service providers. So, I hope on interesting stuff the modern web app development. Welcome Brian.
Brian: Thank you, thank you Michael very much.
Michael: So, let's just start off. People think they probably know what Amazon Web Services is, there's actually more to it than you might think, right. Tell us what it is and why we should be paying attention to it.
Brian: Absolutely, so I think for most developers their introduction to Amazon Web Services was or is a service known as EC-2 which is elastic Compute Cloud. That's their vast array of virtualize servers that come in all sorts of shapes, sizes and memory and speed, and all that stuff. And a lot of people say oh we're going to move stuff out of our data center on the AWS, right. And by that, I mean they're going to run their on demand applications, their custom apps, their licensed apps, inside of the AWS’s computing center, server centers so that they don't have to run their own datacenter because nobody wants to run servers; it's incredibly boring. Well, that is actually the most boring part of AWS.
AWS has expanded so much in the last three, four years into now it's I think – I can't even actually fit all the services on one slide anymore. 02:11 [inaudible] that the slide that any of my presentation with all these rows of boxes and boxes and boxes and boxes. I can't fit all the services on there anymore because they're all about sort of in some ways on demand compute, on demand infrastructure. But they're really about foundational building blocks that can make your life as an application developer, not just a web developer.
But an application developer for mobile, for desktop, for the web and even sort of system applications a whole lot easier and I know I sound like a native US fan boy when I talk about the stuff. I could not build this stuff myself, right. I can tap into it and I can use it and I can provide so much better services for my customers because I'm using it than trying to go out and do it on my own. I mean there's no point in reinventing the wheel when I always do a great shot of the little rickety in the first release. But it provides me with a whole set of services, there's no way I could have the expertise to build or even hire the expertise on my own to build. And so, I pay AWS pretty low prices to tap into all these services, and we get some fun stuff out of it in return.
Michael: And this isn't just one wheel, right? It's like they have like forty different services.
Brian: Exactly, and when I talk about it like I'm going to be talking at C.F. objective. It's really hard to cover everything because you couldn't. I mean they have a conference called ‘reinvent’ that lasts five days and there are over five hundred sessions. And they can't cover everything in those five hundred sessions.
Michael: Five hundred sessions just on Amazon Web Services!
Brian: Absolutely, and that doesn't even cover all the stuff they do. They leave whole sessions, you know whole services out because they're less interesting or they're not the right audience. And the conference isn't saying there's this year, they're expecting forty thousand people. Last year was thirty four thousand people. It's bonkers, it takes over Vegas for a week in November. But it's really fun, and I always learn a lot at that conference which is kind of cool like that. But you know 04:06 [crosstalk]
Michael: I’m just trying to understand how you combine Vegas with AWS. You go to the roulette table and it's running some service or whatever.
Brian: Exactly, exactly. I'm sure some of the casinos have their workloads in AWS as well so it's a little bit crazy.
Michael: I imagine
Brian: But you know, it's fun because 04:25 [crosstalk]
Michael: What are some of the highlights of these different things that are now in AWS that we may not have come across?
Brian: So, I think one thing that I always start talking about because it's easy and accessible to people is S-3 or simple storage service which is this sort of infinite file system for the web, and lots of companies use it. I mean everyone from Instagram, to look our shop. And everything in between uses it to put documents; files, photos, images, videos because it's super-fast and super-cheap. We store about four terabytes of data in S-3. These are just files objects, customer files, our files, content we create. We store about four terabytes and we pay 34 dollars a month for that; something like that. And it's replicated…
Michael: no way!
Brian: … replicated in three different data centers. It's accessible from anywhere in the globe. It's incredibly fast because you're going over Amazon's private network, behind the scenes, out to the public Internet. And a lot of people serve static web sites or regular websites off of S-3 because it's so cheap to do it and to host all that content there. So, I usually talk about that because it's fast, super cheap and it’s there are support for it built into both the ColdFusion engines. But that to be ColdFusion and Lucy have had support for S-3 for years where you just instead of a location on your C-drive, you type in S-3 colon slash slash and the path to your file, and ColdFusion figures the rest of it out.
And all the ColdFusion functions like file related functions work with it out of the box with no modification which is very nice for doing again rapid development which is something that ColdFusion has always excelled at. So, I usually start with S-3 but then, there are a whole bunch of other services. The ones that I talk about are ones that we use because you should talk about stuff that you're doing, right? And what works and what doesn't and so, there's really three ones that we're using right now on top of S-3 on top of the Elastic Compute Cloud. We do some stuff with that as well. And those services are Lambda and Recognition; but that's part of a larger thing I'm going to talk about which is step functions.
And also, dynamo to be which is their no sequel solution. So, those are the ones I talk about here but I could go on and on about a whole bunch of other services that they offer that we don't use a lot of, that we want to use, play around with. If you want to do like 06:54 [Data Lakes] you might use something like red shift or if you want to stream all of our logs to them for real time analysis we could use a tool called 07:01 [Kinesis]. And these are all things we played with but haven't yet developed production apps around.
Well you're just going to have to go to reinvent because I just put the website up and put in the show notes and I discover it's not five hundred sessions, Brian. It’s a thousand
Brian: Is it really?
Michael: Yeah, I don't know how you're going to attend a thousand sessions in a week. You're going to have to multitask yourself.
Brian: My feet hurt so bad last year from walking from the Venetian convention center to the Mirage convention center back and forth, back and forth. That's like a little mile walk from one convention center to the other because it's the ends of these resorts. This year, it's spread between like the Venetian, the Mirage, and all the way down to the 07:46 [inaudible] M.G.M. ground which are like two miles down the strip. So, I don't know how that's going to work out but I can tell you it's not going to be good for my feet.
Michael: I only have one word to say on that’s ‘segue’.
Brian: That’s segue exactly, those of an army of them. For all you register, you take up your segue and that will be forty thousand segues going to the streets of Vegas. That wouldn’t end in disaster.
Michael: No! So, you mentioned you use S-3 in your apps just to be able to store data. What kind of things are you storing in there?
Brian: So, it's really; it's some database although they do actually have tools where you could store a comma separated file in there or log files in there and use their tools. They have a couple ones called 08:33 [Athena] that where you can actually write queries against data stored in sort of structured text files inside of S-3. And some people do that they put like log files in there.
We primarily store fun things like videos and images and Word docs and PowerPoint and Excel and all the sort of user generated content that we generate in our applications or the actions our customers generate in their applications. That's what we store in S-3 because we have to worry about file storage, and we don’t have to worry about replication, and we don't have to worry about speed at which this will be there. And honestly, we can leave files in there for ten or fifteen years if we want.
And one of the cool things about S-3 is they have a limited amount of lifecycle. So policies that you can add. So you can say that after if this if this file hasn't been accessed more than a hundred eighty days, move it off into long term storage where it's not immediate retrieval might take a few hours if you put a request in to get the file back. But, it saves you tons of money. That way, if you have compliance sort of requirements that say yeah, you have to keep this data around for fifteen years or ten years – like we do we have to keep our data around for ten to comply with Student Information laws.
We know it's not accessed but we have to keep it somewhere, and rather than building out file systems that are all equivalent we can say hey, just automatically after one hundred eighty days, move it off into the long term storage so that we don't have to worry about that, and do that work ourselves. So, I mean there's some lifecycle built and it's not really complicated but it's certainly very handy and makes working with S-3 even much more pleasurable; at least for us.
Michael: Great and is that long term storage I'm glad to know or something else?
Brian: Yes, that’s glacier yes.
Michael: So, that basic glacier, yeah. All right well you say glacier. 10:19 [crosstalk]
Brian: That damn hard American accent. Yeah exactly, ‘tomato’, ‘tomato’. My mother says ‘tomato’; that’s because she grew up in South Africa so. And she never got rid of it. She's lived here from year fifty some years and she still says ‘tomato’ and ‘aunt’ instead of ‘aunt’. She rags on us all the time when I say ‘aunt’ or ‘potato’ or ‘tomato’; doesn't like it.
Michael: But at least, I’m saying ‘glacier or ‘glacier’. It basically copies they all fall into tape and then they have a little robot types and loads of back on.
Brian: pretty much yeah
Michael: But you don't have to care about that.
Brian: Exactly, that's the great thing as a thing the reason I like about AWS is you don’t have to worry about what's going on behind the scenes. You just build, and you let them take care of all those details which are knowing and difficult and problematic. And again, I couldn't build a robot army with my limited time, resources and staff. And Amazon can build one. They’ll take it off of you, and put it over there and do all that. They're good at that. I'm not good at that.
Michael: Makes sense to use things like that so.
Brian: yeah
Michael: You mentioned recognition was another service you use.
Brian: yeah
Michael: What does that do for you? Is that this AI thing?
Brian: It is so and recognition is one of Amazon's many 11:34 [inaudible] it's growing army of AI tools. So, they have AI tools that power the Amazon echo so you know we talk to Alexa I should [inaudible] I talk to Alexa right. Whether it's on our phone, or it's on our echo or whatever it is. They have speech recognition services you can tap into. They have text to speech. So, you can say here's a document and read it aloud in the Amazon Alexa voice if you want to do that. And recognition is their image AI, image computer vision sort of tool. I should call computer vision or say it's image recognition services.
So, recognition can do things like you upload a photo, and it can scan that photo for faces which seems pretty straightforward. But it can scan the photo for faces and also tell you basic age range, sex, if they're wearing glasses, if they're happy, if they're sad, sentiment analysis; things like that. It can do face matching and this is what we're actually doing. So, we have requirements about identity verification when students take classes, and we offer lots of classes online and at a distance.
And one of the things the law requires; federal law says the person taking the class or taking an exam is really supposed to be the person who's signed up for the class. Not have some random person take your test, right. I mean you could do a distance nobody would know because you're never in class.
So one of things that we built is a using recognition is a tool that basically when you started exam or started test, it scans your face. You have a video camera. It does basic job using the camera A.P.I., takes a picture of your face and it compares that to an ID that you have had taken at some point the past. Because all of our students have to least have like a picture ID that's on file with the university that your official photo with the university. Usually from something like a driver's license passport; something like that.
And so, we can use recognition to do is say, is this photo that you provided us a long time ago the same person as this photo which is on the camera right now? Because it does image matching and gives you a confidence score that says that anywhere from zero to one hundred percent that these are the same people. And it works surprisingly well. I mean there are certain things it doesn't do so good around. Like if the one photo has no sunglasses and this one has sunglasses, it has a problem with that. But we just say look, you have to have your sunglasses off. But the matching, I've done tests with myself for example where it's a picture for me from fifteen years ago and today and it…
Michael: really?
Brian: Yeah, it's kind of cool and I've done it with like my kids where pictures from them five years ago versus today. It's like yeah, it's the same person. It's really quite good at that. So, that's what we're using it for. But there's another feature you can use it for is for a label; labeling images. So if you wanted to… they actually recognition for Amazon's own cloud storage search. So what if you store your photos on Amazon cloud storage or photo storage.
Michael: And I believe it's called, right?
Brian: Right, right. No, I think it's Google. I think Google's 14:35 [precursor] I think.
Michael: Oh okay. It has the same feature in Google.
Brian: Right exactly so it's the same thing in Amazon photos. Thank you Michael that's exactly what it is. So, if you search for dog, it will use the same technology to scan through all those photos and look for images with a label of dog, and you can tap into this for yourself. So, if you want to do an inventory of clothing you could say you know, instead of someone manually going through and saying labeling all the colors of the pieces of clothing, or the style or whatever.
You could tap into Amazon system and there are many, many, many, many labels to come up with a classification for that. So, it's actually quite powerful, it's quite fast, and really quite again fun and easy to use using… For all these things we use there are Java S.T.K. So everyone knows who works in CFML it's built on top of Java. So we drop the AWS, S.T.K. for Java into our live directory in our CFML engine, and we have access to all those services directly from ColdFusion.
It's more Java like, right. It’s not nicely wrapped in tags but it's very easy to read, very easy to use. It takes very little time to rip out these services and rip out these new features because we're using all that stuff that's been done for us in the Java S.T.K. from the get go. Same thing works for Python. Python has a very robust library called ‘Voto’. That you can use if you're a Python developers as well.
Michael: So, you mentioned also you’re using reflector for image, right. So, what is that doing?
Brian: Right, that's the image library that I was just talking about. Reflector is the name of the image library.
Michael: Oh! Okay
Brian: I don't know. I guess reflection has to do with images and stuff like that. So, they have three main libraries for AI. 16:25 [Pauli] which is the text to speech reflector which is image recognition, and search, and sort of data labeling. And then 16:34 [LAX] which is their natural language processing that they use for Alexa, and the Amazon echo. And those are all available to developers. We're only using recognition right now.
Michael: Wow! So if you wanted to create an Alexa you could effectively code it using all this stuff.
Brian: Pretty much; you'd use the Lax library to and that if you can actually go out and do this now if you wanted to. They have lots of tutorials on their website to talk about like how do you set up a chat bot? How do you set up voice recognition? So, it like looks for a very specific tool.
Like they call intense and slots for you know. An intent would be something like order a cup of coffee and a slot would be the size of the coffee cup and all the different options that go in there. And that's all stuff that they have on there right now on the website. We just haven't found an opportunity yet to use this in a real production app.
Michael: Yeah, my mind boggles a thought of having apps…
Brian: Having that much power, right? there's a really thinks well if you
Michael: 17:35 [crosstalk] yeah
Brian: Yeah and you don't have to worry. The great thing again, you don't have to worry about the infrastructure. It's just there and you can build your apps that you need to build on top of it knowing that the underlying technology just works and is there for you.
Michael: Unless they have one of those outages 17:54 [crosstalk]
Brian: Yes, the great S-3 outage of February 2017 which I will talk about in my presentation. Yes, so that is one of the challenges with the on AWS or I.B.M. blue mix or Microsoft [inaudible] services or Google Cloud platform is that stuff goes wrong. Once in a while there are outages.
So we've been using AWS for a long time. Like I think it's been about six years now at least our time with S-3 and then moving on to other services. And the outages are much less common than they used to be. But stuff still happens and that's something you have to plan for. You have to have a strategy for it. But the real question I think depending on the size and scope of your business is that people talk about redundancy in multiple cloud platforms and all that good stuff is, what's the loss going to be? What's the cost going to be for you to have down time because of these cloud services that might happen? In this case that was the first outage in S-3 and almost four years.
So, do you want to pay for the cost of rebuilding everything another provider, having redundancy in multiple or Amazon regions around the globe; whatever it is. And all that cost for an outage that may last two or three hours on one day every four years. And I think that's the sort of you – people get freaked out about the cloud and what it might cost. But I think that's the real business question that has to be asked when you're looking at cloud providers, and redundancy, and dealing with outages.
This is what's the real actual cost your business? For an Amazon itself right, for a shopping website that does super high transaction volume it's not acceptable. And but you probably in those cases have the resources to do all this work, right. This redundancy.
Michael: Doesn't Amazon need their own dog food? I mean don't they use S-3 themselves. So, when that went down they were screwed.
Brian: Right exactly so. Every single service that is provided, Amazon has built and used for themselves internally first. So, they use every single one of these services. Like I said, the Echo the Amazon echo and dot, the Alexa service is powered by Amazon Lex and Amazon Polly. So these are services they've already been using for quite some time in the background before they let us do it.
So, they use S-3, and they use dynamo DB, and they use lambda, and they use cloud front; which is their global content distribution network. They use 20:25 [Kinesis] says for real time data streams and they use 20:27 [Alast] search for doing internal search of unstructured data. They do all of these things themselves internally and then they make it available to the public for you. So yeah, they dog through their own stuff in a very, very serious way.
Michael: Yeah, so that gives me a lot more confidence than any promise someone can make in S.L.A.
Brian: Right absolutely, like their Amazon. If SLA goes down like you said they're screwed. They're going to lose a lot of money too.
Michael: So, you mentioned creating micro services that you call from ColdFusion or invoking 21:01 [inaudible] no J.S. to run those micro services. Tell us a bit more about what you’ll be demoing there.
Brian: Sure; So one of the – as a C.F.M.L. developer, one of the things I think you should try to do, that you need to be doing to be a better develop over time is use other languages, right. Whether it's JavaScript, whether it's Python, whether it's Dot net; whatever it is you're going to use something other than pure C.F. Even if it’s Java and Dot jumping on the job, use something else.
And when the node bandwagon started blowing up a couple years ago, everyone’s like no, it’s the best thing ever, get rid of C.F. It was like well no, we're not going to get rid of our investments in C.F. because it's huge and important to us and C.F. actually makes us productive people, productive developers.
But we want to try out this node thing. We can see where there is value, we can see the high IO throw but now there are things… and even if it’s just to play around with it; cool, great. You know around the same time as node started coming on the rise is also the rise of micro services, right.
Everyone’s like get rid of your 22:00 [mile monolith]. Everything in micro services no matter what, regardless of what your architecture is. And that's great for Uber, Netflix and everyone else. But it doesn't quite work that way and for many people, the journey of micro services is a gradual and progressive one and should be one.
And AWS came out with this system, this tool called ‘Lambda’, the service called Lambda. It was the sort of the first server lists, right. Functions as a service. It was the birth of the server less movement in a very real way where the idea was there's not even servers you worry about. So with easy to, you have to manage to maintain a server; whether it's a Windows or Linux server you have to maintain that. With Lambda, you just write code that just magically runs, right. So what's actually happen in the background is yes there are servers, there spinning up Docker containers with node draw on times. It’s where your code is being dynamically loaded in.
Yes, there are servers but you don't worry about it. You just write functions, and the functions respond to requests, and they send stuff back, and respond to request. You don't worry about server so, it's really cool. So, if you need to invoke the same function on ten thousand images, you have to build up a task management queuing system to do that. You have to build up a farm of servers to do that.
You just say here's ten thousand images go, and Lambda handles the rest of it. And we use lambda for a number of different things. For probably the most interesting thing we use it for is a whole automated video, encoding, transcoding, and transcription workflow. So, I work for John’s Hopkins public health; we're an education institution. We do a lot of video in our classes and this video has to be delivered. As we all know, on the Web you can’t just put up a MP4 file. It's got to be in different sizes, in different bit rates where them for people who don't have MP4 support on their Android phones, and we've got to caption and transcribed these things for accessibility reasons, for universal design really reasons really.
And we could do all that by hand and that would be terrible. But, so what we're using instead is we're using Lambda as a workflow center, workflow hub. So we wrote these scripts that are all tied together on Lambda, so we drop off a video on S-3, Lambda listens for those events, runs some code that some prep work, IDs that tags it, stores that information and Dynamo D.B. which is our no sequel database. And then, also calls another service called ‘Elastic Transcoder’.
An elastic transcoder is a service they've used for a number of years, it's the heart of Amazon video that takes video and transcodes it into all sorts of formats or audio and transcodes it to all sorts of formats. And it transcodes our source video into all the formats we need. Each of those get dumped off back into S-3 at the same Lambda function listens for that and says oh, this is a drop off, let me update this information, the database table with all the different versions are all done, Lambda nodes for this, listens for this, checks it, marks it off from the database, talks back to our ColdFusion application through a web hook. And says hey, this is now publishing available for students to consume.
And all we have to do, my team has to do is literally move that file into a launching point in S-3 and we're done or our application can do; that actually what we have been doing. Somebody uploads a file to the app, all that another work which can take hours and hours and hours to do because video transcoding can be a very time consuming and intensive process.
We don't worry about it. It's all handled through this are and we don't run servers. We don't run any video encoding servers or anything like that. We just say, ‘Lambda do your magic’ and that's it. And the code for wasn't really difficult to write. It was all written in node using existing libraries the A W S S T K for java script which runs inside of node and some other libraries there. Promises to make our life easier because promises always make life easier in programming.
And it was great, it didn't happen overnight, and it's probably the most complicated example of what we do in Lambda. But the great thing about lambda is that A.) we don't run servers B.) any kind of like asynchronous batch processing task thing that we want to do we're really moving to Lambda now because it let's us build a node, it lets us work in that environment. It's really easy for ColdFusion to talk to Lambda, and to launch these services, and get data back through the A.W.S.T.K. for Java. So, it's really nice because now, we're both node developers and C F M L developers without running a single node server ourselves, and that’s one of the really cool things about Lambda.
Michael: That is cool. So this takes the whole Docker revolution to another level.
Brian: absolutely
Michael: 26:33 [Inaudible] have to provision the Docker containers.
Brian: Right, lambda does all that for you. You just plug your code in there and you're done and it's pretty darn cool.
Michael: And I'm wondering what the next step in this progression is. No servers, no containers.
Brian: No functions, right?
Michael: Yeah exactly, I'm wondering if you just tell it. You tell Echo, Amazon Lex or whatever you know I will not function for this and it goes off and writes the function and runs it in Lambda.
Brian: I mean with AI. AI is writing its own programs, its own languages. That may not be too far out possibility.
Michael: Yes, maybe the movie Terminator had it wrong. Maybe it won't be a violent and maybe it'll just be a lot of software being developed.
Brian: That's right, exactly. We’re all to sit back and well, maybe it's more like Wall-E, right. Well just get really big and fat and from not having to do anything because it takes care of everything for us. One of our giant sitting apps. That will be great.
Michael: So you also mentioned you use a node sequel database. I guess I don’t know if database is the right word for that; data store.
Brian: Document data store.
Michael: Dynamo DB that you; Yeah, document data store.
Brian: right
Michael: What are you using that for? Tell us about how you do that.
Brian: Well so again. So Dynamo DB is AWS’s no single data store and instead of running our own Mongo cluster, or Couch G.B. or something like that. Again, because I'm not big into running servers. Running servers is not fun at least not for me and my team. And we don't have the resources to really do it. But again, there are use cases in which no single databases are great.
I'm not a huge fan of no single databases in part because I think they're over used for things that certainly modern relational databases could handle much better thing. Like 28:26 [inaudible] I think is awesome at so many things that people try to use something like Mongo for especially with it's Jaison. It's native Jaison support. Now, I think sequel server 2016 has native Jaison support as well. So, not always the best uses for it.
But what we're using a Dynamo D.B. for is for capturing data streams; unstructured data. So for one example of how we are doing this is that we do data analysis of what students are doing when they're taking exams to see like how much scrolling are they doing, how often do they change their answers? Depending on the length of the exam; how long do they generally average stay on the page. And looking at the data the clicks that come in, the scrolls that came in; all that sort of event data from the rouser. All that event JavaScript base sort of event browser data. What scrolling X. Y. locations, what they clicked on, what all that good stuff. An additional date on top of that like their IDs and what question ID, and what answer ID, and all that fun stuff.
So we can do analysis on this and see what are the real behavior patterns within a given course, or across courses when students are taking exams because this is information faculty will want to know. But as we're doing this, we're like well, attritional relational database won't be a good data store for this, right. Because we're going to be generating way too many events way too fast.
All those inserts would just wind up killing a single table. And if you're trying to join across tables, we have to know or document structure ahead of time. And a lot of times when we're doing this kind of research work, we don't know all the attributes you want to include. And so a document data store is great for that because it's flexible, right.
We can say the first million requests that came in didn't have attributes X. Y. and Z. But the second ten million or the next ten million did have attributes X. Y. and Z. but we dropped A and B because they weren't really interesting to us. So, we don't use dynamos a long term data store. It's really like a good place for us to do some basic querying, some basic capturing of data.
I think that it's possible I mentioned earlier on there's a tool called Athena that can query C.S.V. files. And it's possible we may shift into doing more of that in C.S.V. files. But again, C.S.V. files have to be structured in some way so that the querying engine knows that the third column is the country column. Or the fifth column is the date on which this event happened. It's nice to have unstructured data, and nice to have that flexibility. And for us, that's really where dynamo comes in; is when we need to capture data in a not so structured way.
Oh yeah, we still use relational databases will never; I can’t say never. But probably never stop using them.
But dynamo D.V. is another nice add on on top of that. And what's really nice is it's extremely high theropods. You can literally do you know, if you want to hundreds of thousands of inserts per second. We don't pay for that because that's way too expensive. But for our purposes, it's pretty very fast and very flexible and again, we're not running our own servers. We use the same S.T.K. that we use for everything else. All the other work that we use rather than learning a new set of technologies and a new set of S.T.K.’s to access that content.
Michael: So basically, you decided to use Dynamo DB instead of Mongo DB because it was part of AWS and it was all integrated in.
Brian: Yes, pretty much. I mean that's sort of the vendor lock and people keep warning you about right. But for us, there was no compelling reason to set up our own Mongo cluster or even to use the very many services that are out there that provide that service for you. It's like look, we're already doing it inside of our C F M L op or are using a WS has DK. We're familiar with our mission’s model, let's go and use that.
It's a very simple tool compared to some of the things you can do in Mongo. It has its own version of like the abbreviations framework and you can do some somewhat advanced clearing, not quite like Mongo. But then again, the days that were not there were putting in there right now for us is not so mission critical that we need incredibly robust aggregation frameworks or anything like that to query against it.
Michael: And is it distributed as well if you want to have redundancy?
Brian: Yeah, redundancy is already built in and you can do across what they call region replication. So, AWS has now I think it's gosh I want to say eighteen data centers around the globe, and within each data center they all regions. You know geographically distributed all over the world. Within each reason, they have actually have three physical data centers.
So for example US East is the first one they set up. It's in Northern Virginia but, it's not one building. It is three separate buildings within about five miles of each other. That is all considered to be one region. So when you say and so each building was what they call an availability zone.
So within two like Dynamo, you automatically get replication across availability zones. Meaning in case one building were to blow up, or one building were to lose power, your data would automatically replicate in those two other buildings which are about five miles apart. If you want to go one step further, you can replicate data across regions so that if you have lots of people in China, they're accessing the China region rather than the Virginia region.
It costs a lot more, there's a lot more complexity in terms of that set up because you're basically doubling all of your costs, doubling your use of all the services. But it is certainly an option and companies like Netflix. The reason why they're so amazing, and widespread, and truly global is because they use data centers in every single one of AWS’s regions plus their own sort of hardware and ISP’s around the globe.
Michael: It's mind boggling how much time and money that must a cost to build those data centers. I mean just building one data center is an enormously lot of work.
Brian: Right and yeah and it's great to have that capacity. I just can't even imagine just the effort, the ongoing effort of building and maintaining. And then, when you push out a software deployment and update, right. We're going to update a course feature of S-3, or an update of course feature of dynamo DB. How kind of terrifying that must still be. Even with all their automated systems and their testing and their great development practices. How terrifying must it be when like literally half the Internet relies on a service like S-3, right?
What if something goes wrong? You have just broken half Internet because you did something wrong which is pretty much what happened back in February of this year. A lead System Administrator literally fat fingered the number of indexes or so as to be dropped and rebuilt. And instead of like 34:55 [one hundred], it turned into like ten thousand. And it completely brought down S-3 which brought down almost half of Amazon and AWS in the process I would not want to be that person. It would not be fun.
Michael: No, definitely not. So, what are common problems that come up with all this cloud stuff when you're doing programming with it? And how can you avoid them?
Brian: So, I talk about this a little bit in the presentation. The common problems really resolve around failure, right. You can talk about vendor lock in. I don't really believe all that much in that argument although, some people certainly do from a very strong kind of moral philosophical point of view.
I think the biggest issue going into is failure. What happens when things go wrong? What happens when S-3 goes out? What happens when Dynamo D.V. goes out? Now, these don't happen often. The S-3 outage that happened back in February was the first time in four years that S-3 had any sort of serious major or extended service problem.
But you have to have a plan. You know that plan can just be blame Amazon and be like head in the sand or blame Amazon. It's their fault. It's what a lot of people have to do is cloud service providers right. You have no other choice. As you know building that kind of replication or building a backup system can be incredibly expensive, right.
You recently go to Amazon, you go to Google, you go to Microsoft because they've built these systems already. You don't have to make that investment. So it could be just blame Amazon, or it could be that you should think about better caching strategies right; back pressure. If you notice you're paying a service and it's down then you say okay, we're going to shut off these features of these services, or it is going to pull everything from the cache rather than pulling the live real data.
Can your business handle that? Can you turned off ordering for an hour or two when this goes on? These are all questions you have to ask yourself and it's really just about having that plan and understanding the tradeoff between full 99.99999999999 percent up time and the costs of that kind of up time. Because for most people and most businesses, that cost is simply too great and too significant. Considering that outages are really; at least on AWS quite, quite unusual.
Michael: So that's kind of a friendly degrading of the functionality in your app.
Brian: right
Michael: You know maybe the image recognition part doesn’t work but other bits continue to work.
Brian: Absolutely, absolutely. I mean imagine have local copies of your data, your core data, your caching things locally. If you're using e-tags on the images, on your pages, they also can be cached on the project for quite some time without ever having to go back to S-3, or cloud front or any of their services. So yeah, it's really just being a little bit smarter about how you look at either caching or degrading the services that you offer on your website.
Michael: Anything else you want to share with listeners about using AWS or other cloud stuff from ColdFusion?
Brian: You know I think one of the things that keeps people away from doing this is that it sounds scary and difficult; it's not. Like I said I think AWS and even other use and super cool cloud before; it's simple. The STK’s they provide are super simple and I know that people sometimes get freaked out because it's like oh it's the S.T.K. for Java. I have to write Java, I have to know about Java.
Well, if you know a little bit about how objects work in 38:11 [inaudible] languages like Java, if you know how CFC’s work, you can pretty much figure out how the whole Java S.T.K. Every method is very clearly explained. In the docs it says, this is what's expected, this is what it returns and C.F. dump is your friend. Yeah, 38:27 [right dump] is your friend right. If you don't know what the methods are in the object you write out like oh, this is what I'm looking for and this is what this method is looking for. And yes, it can be a little verbose sometimes.
But again, the power that it provides you just because something's a little complicated to learn at first I wouldn’t even say this is all that complicated. It's pretty straightforward if you know how to read the docs, you can figure it out. And there's lots of examples on their blog, and in stack overflow because people use the Java S.T.K. for AWS a lot. Just because it's a little different, just because it is Java doesn't mean you shouldn't use it because the benefits are so powerful and so enormous. And what you can do with that.
I mean like again, the thought that we can talk to Lambda, and we can write stuff in JavaScript and Lambda and talk to it from C.F. with so few lines of code. I mean it's like five lines of code to invoke the function, and to get it working, and get the data back from AWS, and have this whole infrastructure in the background that we never had to build. And we can just tap into is so great and liberating, and it's fun; that's the other thing.
Is sometimes, development isn't fun right, and people make the argument that ColdFusion isn't fun because it's old language. It's not as fun as node. Well you know what? If you tap into a service like Lambda or if you tap into open 39:43 [Wisc] on I.B.M. or Google cloud functions and they all run node, you can have that fun. You can do that fun and discover the joy of polyglot programming, and staying current, and doing new things. And I think that's the thing that I hope people will walk away from my presentation with. This is some sort of inspiration going out and trying it out because it's not hard, and it's really fun, and incredibly powerful at the same time.
Michael: Great! Well, good luck with your presentation at C.F. objective. I'm going to change gears a little bit.
Brian: sure
Michael: Yes so, why are you proud to use ColdFusion?
Brian: So, I love the fact that I can be super productive in ColdFusion in a very short amount of time. So for example, I had to – I get requests for lots of data and I teach a class here at John’s Hopkins which is a prerequisite class to all the incoming students of the School of Public Health. And people keep sending these Excel files and they're like okay, can you tell me of these students took your class? You know if these students… It's a pain in my butt. It's tedious, it's boring, it's repetitive and the spreadsheets are all different and are never quite the same and I'm like okay, fine.
What I'm going to finally just write a tool that will let me read the spreadsheet regardless the columns. Match on various columns blah, blah, blah. blah. You know, I knock the thing out in about five hours because including all the testing and everything because of C.F. spreadsheet right. That functionality is built into ColdFusion. I have to go find a library, figure out the library, add it all in, and it does it still supported does it work with J.D. K 1.8 versus 1.7. I don't have to worry about all the stuff. I can be super-productive, super-fast.
That's one of the reasons why I'm proud to C.F. is because it makes me a more productive developer. And when I want to go outside of C.F., it's really easy for me to have C.F. talk to something else; regardless of what it is. For us right now, it's a lot of node apps, node services you know C.F. talks HTTP. HTTP is the lingua franca of the web. It's lingua franca of Amazon Web Services. Amazon Web Services everything you do is HTTP request. Even if using the AWS S.T.K. for Java. It's making H.T.T.P. requests in the background; ColdFusion’s part of that.
And I can be super-productive and I can talk to there's a languages and be a polyglot programmer because I don't have to spend tons of time. When it comes to fun stuff like a spreadsheet, or P.D.F. generation, dealing with horrible libraries, I can just make it happen within C.F. within a couple of lines of code. I love that.
Michael: That is great. So, sometimes people read the ColdFusion is dying and that blog posts has being floating around in the papers for so many years. It's ridiculous because obviously, it hasn't died.
Brian: right
Michael: But what would it take to make ColdFusion more alive this year?
Brian: So, that's a really good question. I mean aside from people being out there and saying hey, I'm using ColdFusion with this thing. I'm doing this in ColdFusion these cool projects. I'd love to see more projects highlighted by Adobe and Lucy. I know that their time is limited, and especially Lucy since it's no open source product.
Who's really going to be an evangelist for them when they don't have full time staff necessarily to do it? Although, I think the 42:55 [inaudible] team does an amazing job of evangelizing C.F.M.L. They really, really do and I'm very thankful for everything that Louis and Brad and Gavin and Eric and everyone does who's on the team. Because they really bust their humps trying to promote C.F.M.L.
But I would love to see more marketing, or highlighting of what you can do with C.F.; I really would. I think people think it's a dead and dying language but down, people are saying oh, Ruby on Rails is dead and dying. Why are you using that? It's so old, it's from two years ago. When two years ago, it's like here's a ColdFusion you should be using Rails. It's the best thing ever and now, it's Express and it’s node and whatever else.
So, from a developer perspective, I would love to see better efficient official tooling support. I mean I know there are certainly add ons that are out there. Certainly the [inaudible] team does a lot with test box which is critical and important. But I feel like Adobe could do better with the tooling around C.F. itself.
A language I think is pretty mature, the language is fine the way it is. Especially in C.F. scripts and tax I think by now and 2016 they've worked out a lot of those issues at least Adobe has and certainly Lucy has. I would like to see better tooling and support. I would like to see the app server sped up in like two seconds. Like the node environment can spin up in like two seconds, or Ruby environment, or Rails environment it's been up in a matter of seconds.
Doing testing and waiting for everything to stand up is one of my big developer time blocks. But again, people see that and it's like well the ColdFusion app Server takes thirty seconds to start up, when I do something in node, takes two. That adds up over time and gives a perception about what ColdFusion is and isn't, and is a dead and dying language because it takes so long to start up.
So, addressing some of those tooling issues at the core would make it better I think for the entire community. And make people say yeah, you know what? This is an equally viable language in the same way that JavaScript, Flash node is as well as Ruby, as well as Python, as well as anything else.
Michael: Those are some great suggestions for making ColdFusion more alive this year and that's part of my mission with this podcast. You know, get people infused about it and seeing ColdFusion is a modern language if you use it right.
Brian: absolutely
Michael: Just to address the faster service spin up. I was in a discussion with 45:17… into the box who is one of the Lucy folks. He's been working on speeding up the spin up time. I don't know what number of seconds he's got it down to. I think he started at eight seconds and he was trying to improve on that. Whether he got it to two seconds, I don't know.
Brian: That's pretty amazing.
Michael: Yeah because if you're using command box or some of the way to just cool stuff, you need it to spin up fast.
Brian: absolutely, absolutely
Michael: So with C.F. objective coming up in just a few days now, what are you looking forward to at the conference?
Brian: So No. 1) Here's what I'm looking forward to. I’m looking forward to the speakers because I know quite a few of them and they're wonderful men and women, and they're cool, and I like hanging out with them. And I like to harass Gavin 46:03 [inaudible] and among other people; they're great.
They're also a lot smarter than me. The thing that I love about C.F. objective and the reason why I've gone to the conference; I don’t know how many years in a row now maybe this is like my eighth. If this is the eighth C.F. I think I missed the first one. Is because there are people who are so much smarter than me, and I always learn so much from them as individuals in their sessions.
One of things I like about C.F. objective is that it's C F M L sort of oriented but there's a lot more than that, right. There's a lot of stuff about the front end, there's stuff about Docker, there stuff about graph S.Q.L. 46:38 [inaudible] excuse me which I'm looking forward to learning more about from some really smart, really good presenters. And I've seen these people present before many times. They're super smart people and I always feel like even as someone who's been doing this for a long time and it gets to present too, I learned a lot. And they’re approachable people; that's a really nice thing. You know talked about the AWS conference the beginning. It's hard to network with 40,000 people, right. Even with a two thousand people.
Michael: just slightly
Brian: Just lightly; But the conference size of C.F. objective, I can meet and I can talk with people. I can see people in multiple sessions on the same topic or different topics, and we can talk, or we can have lunch together. It's much easier to network in that smaller environment particularly with the other speakers.
And I think that's one of the great advantages of the small conference is you really do get those opportunities even if you are not the best sort of most self-promoting networker that's out there. Even if you're a pretty sort of introverted person, you can to sit down next to somebody and strike up a conversation because you have a shared topic to talk about not the sessions that you saw that day.
Michael: And if that doesn't work, you can always do it over the networking events.
Brian: That's right, there are those as well, exactly.
Michael: So, if people wanted to find you online Brian, what's the best ways to do that?
Brian: Well, probably Twitter and GitHub. So I'm at github/brianklass; all one word. On Twitter, I am Brian_Klass. If you tweet at Brian Klass, he is a political professor over in London who's my doppelganger apparently now. Who's tweets now somehow he bashes Trump all the time. He's left leaning, he's very outspoken critic of Trump.
And so, it's very bizarre in Twitter time line. My name just keeps reappearing and reappearing because other people that I follow, follow him and re-twit his stuff. And I'm like no, I was there first. I just accidentally put 48:33 [inaudible] in my name. I don’t know why I did that. I never should've done it. But yeah so, Brian_Klass on Twitter, and then brianklass; all lowercase, all one word on GitHub are probably the best ways to see my work and of course, to get in touch with me.
Michael: Great! Well, thanks so much for being on the podcast, Brian.
Brian: Thank you Michael very much. It was great. I had a great time.
