You can listen to the podcast and read the show notes here
Michael: Welcome back to the show. I'm here with Scott Caldwell from Computer Know How and he is going to be talking about managing an international team using Git, ColdFusion, Node, Joomla, and all the headaches and heartaches he went through on a big project. So we're gonna look at how he used source control. In particular, Git and why that's critical for all ColdFusion developers.
How to best manage lots of incoming changes by Git, and keeping aligned with your business process and release cycle at the same time. Lessons from the trenches on working with multiple teams, in multiple time zones, in multiple countries and multiple programming languages. I can see where the headache and heartache part came in in this talk. And avoiding deployment day disasters and project management war stories. So, welcome Scott.
Scott: Thank you so much for having me, it's an honor to be here.
Michael: Yeah, great to have you on. So I know in our recent state of the union ColdFusion survey, not everyone uses source control. Maybe about 20% of people who took the survey don't use it, and of course we all know that only the most engaged developers take online surveys 'cause they see them in forums or what have you, so I've [crosstalk 00:01:14] gotta guess that it easily could be that 50% of program developers out in the wild don't use source control. So why in your opinion is that critical that they use it.
Scott: Oh man, you know developers are so anxious to get started on things and they just jump right in and sometimes source control is seen as a second-rate thing and, but I gotta tell you it's just crazy not to use it. It is absolutely critical. Even for a one-man show to have things in source control, to have a good history of what you've done, and to have those backups and the possibility to deploy in an automated way. I just can't imagine not having source control on every single project that I work on.
Michael: Yeah because it lets you roll back to an earlier version in case the deployment goes bad, and it even lets you have multiple different versions all in the same code base.
Michael: And that's with a single developer. If you have multiple developers in multiple teams across multiple countries, I can't even imagine how you do that without source control.
Scott: Absolutely, you've got so many changes coming in the door and trying to get out the door in an organized fashion, and if you don't have a good source control strategy then it's chaos, absolute chaos. And I think we'll talk about that more in a little bit but when you've got more than one person going, you have to have a system in place to manage all of those changes or your life is going to be crazy.
Michael: So what is your favorite source control system and why?
Scott: You know I haven't played with all of them and I know there's some great ones out there that I haven't touched. I've had … Most of my experience has been in Subversion and that's kinda what I cut my teeth on. I joined a team right after they switched to, or right as they were switching from CVS to subversion and that's where I really learned it. And then I was able to … A few years ago, I guess about four years ago now, we switched to Git from Subversion and really haven't looked back.
You know, I've got to tell you when I first started looking at Git I thought, “Why, what's the point? I don't understand what all the excitement is about Git and why is it so much better and this and that.” And now that I'm in it day in and day out and I go back to a Subversion project it drives me bananas.
It's night and day, and so I think Git is definitely my tool of choice right now. Again, there's a lot of great ones out there. Mercurial I hear is really good and so, but Git is the one that I've had the pleasure of working with the most and definitely the one that I choose right now.
Michael: Great, so what exactly is the difference for you? When you said Subversion now drives you bananas, what exactly about it does that?
Scott: You know, some of the creature comforts that Git provides is just really great in terms of stashing and being able to commit but not push, and some of these thing really make my development flow a lot smoother. And then certainly some of the grace with which Git handles branches and merging and things like that is definitely a step ahead of where Subversion was, at least when I stopped using it. And so with Subversion it was a little bit scary anytime you do a merge and with Git it's, you know, you don't even think about it anymore it just works.
Michael: So you mentioned Git stashing, for people who haven't done that, what does that mean?
Scott: Git stashing is almost like doing a commit but only on your machine and kinda off to the side. So if I'm working on something and suddenly a critical issue comes in that I need to fix right away, I can stash whatever I was working on, set it aside in kind of a special place, go fix whatever I needed to fix and then bring back my stash with those changes and get back to where I was.
And so it's kind of a pseudo commit but it doesn't get recorded into history, it doesn't get pushed to any servers or anything, it's just a way to take my working copy and set it aside while I work on something else.
Michael: Cool, well that sounds really handy when you're working on a fast-changing project.
Scott: It really is, yeah.
Michael: So when you have a lot of changes coming in [inaudible 00:05:29] from a team that's spread all over the place, tell us how you keep that aligned with processing those changes and aligning it with your business process and release cycle.
Scott: Yeah, so that's really the trick of it all is coming up with a strategy to manage all of that and to organize it in such a way that you can get exactly what you want out the door, be able to manage the things that really shouldn't go out the door quite yet, and sometimes you want to work on those longer-term projects that are going to be around awhile but if they leak out early they could cause some problems. And so when you're considering your strategy you do have to take into account what are the business processes, what ticketing system are you using, and what do your product managers and upper management expect in terms of being able to hand pick which features are going to be deployed and whatnot.
And so there's a lot of considerations to take into account. We've looked at a couple of different flows and different processes for managing our repos, and we've kind of oscillated between a couple of different things. One that's super popular that we've spent quite a bit of time in and gone back and forth with is GitFlow. It's popular enough that it's built into most of the modern tools. They have specific, you know, you have your Git buttons and everything on your tool and then you'll have your GitFlow buttons, they just build it right into the UI. And so we've definitely use that.
We found some challenges with that in our particular organization and with the way our business works especially, and so we had to look for a couple of other options. And we found another one called Branch Per Feature that has worked really well for us and has given us the granularity of control that we needed to manage all of the different things and to be able to say, “That one's going, that one's not, this one's going and whoops that one has a bug let's pull it out last minute and everything else can go.” But and so our management required that level of granularity and so we had to find a git process that matched that.
And so again there's a lot of good ones out there. GitFlow is definitely fantastic and every time we have any challenge with Branch Per Feature, we go back to well maybe we should look at GitFlow. And maybe we will eventually, and that'll be another war story, but right now Branch Per Feature's working pretty well for us.
Michael: And both of those can help you keep track of bugs and change requests, is that?
Scott: Absolutely, so they allow you to make code changes on branches for individual features or bug fixes or hot fixes. And they give you the opportunity to pick which ones are actually gonna be put into the release. And so that way, if I fix a bug obviously that needs to go in, but if I'm working on a feature that I'm halfway through or maybe I submitted it for review and they wanted a few more changes and that kind of thing, then we can hold that back and set it up for the next deployment instead of forcing it out the door right now because, well, it's deployment day and we have to deploy. So both of those give us the opportunity to control what's going in the deployment and give upper management the peace of mind that we're not sneaking anything in.
Michael: Cool, so what kind of release cycle do you work on? How often are your releases?
Scott: Yeah, so again with that we've gone back and forth with different processes and I know every company, every single company is different. The one that I work most closely with right now is a three week cycle. It's two weeks dedicated to development, one week dedicated to a QA cycle where we get validation on what we're deploying and then also an opportunity for bug fixes and defects and things like that. And then also during that time we're kind of preparing for the next release. So that's given us a good cadence of a regular release, a good solid block of time for development, and then a week of QA and planning that sets us up for the next block of development. So right now that's what we're doing. It'll probably change next week and but right now it's given us a pretty good cadence.
Michael: Great, and so when you say the word cadence there, you're saying how often new features make it in, or?
Scott: Yeah, how often we deploy ultimately, aside from the super urgent bug fixes. We want to have a cadence not only for, or regular deployment cycle not only for the developer but for everyone up the chain. Because our development and the releases don't just affect the developers, it affects the project managers, it affects the product owners and everybody up the chain, and they need to know and communicate what we're doing.
And so if I turn something, you know, the header from green to blue, it's not just about me getting my code out there it's about having the release notes written up, it's about communicating company wide that this new change is coming. And so if we have kind of a regular cycle for that, then that helps everyone kind of get on board, figure out what their pace needs to be to keep up, and to know when to expect their bug fix to get released. We can say, “You know what? That thing's really important. We'll have it out in three weeks” because that's what the beginning of a development cycle. Or we can say it's going to be 12 weeks because it's not as important and we have bigger things going on, so it gives us a way to communicate and be on the same page.
Michael: Great, so I'm guessing you're using some kind of agile methodology here, is that right?
Scott: We are, yeah. You picked up on that and we do claim to be an agile shop. And we do our best and are ever aiming to get better at being agile and-
Michael: Which parts of the agile process are you using and which don't you like?
Scott: I'd say we're doing pretty good right now at being agile. We actually had some consultation and some coaching in that area and so we have all the meetings set up the way we're supposed to, I think. And we have daily scrums and we have a really good team communication. We still have some challenges in terms of getting tickets defined all the way before we start working and reprioritizing in a way that is always easy to follow, but by and large I feel like we are following the agile process as much as I understand it.
Michael: Great, and how does that compare to what you guys doing before?
Scott: Before it was very ad hoc and it's been part of the process for a growing team as well. So before there was just a couple of developers and really we would push same day when we would fix something. It was like “Hey, my thing's done, I'm gonna push it out to prod.” And so we definitely have matured or, depending on how you're looking at it, regressed as we've grown and tried to establish more process around it all. But before it really was, we would just talk all the time it's be like, “Hey, I'm working on this and I need help” or, “This has been a challenge.”
And we never called it agile but we've tried to be agile in the sense that it means flexible and adaptable. And just push things out the doors as we needed to. But over time we realized we can't just, we're not keeping up with everything that's coming in the door. We're losing things. Somebody said that this feature was really important and we completely dropped the ball because some other thing came up and we forgot about it.
And so putting more of again that cadence in place and having people kind of watching that process and in terms of BA's and whatnot, scrum masters. That really helped make sure we didn't drop any balls and to make sure that we were releasing things on time.
Michael: And the scrum master is basically facilitating all the developers, helping them out and solving problems and communicating well. So it's similar to a project manager but they're not kind of dictating what you do, just more team communication?
Scott: Yeah, they greatly facilitate that team communication. They somewhat of a taskmaster in terms of just making sure that you're working on what you're saying that you're working on. And they're looking at the kind of the whole sprint or release cycle and going, “Okay, this is what we've said we're going to get done. How are we doing on getting it done? Is this really achievable, or do we need to go back and tell somebody that maybe this feature is going to get delayed or whatnot?”
So it's really good just have somebody that's always there coordinating, arranging people and tasks and making sure that a developer has all the information he needs in terms of requirements, and all the tools that he needs. And so it's definitely a very valuable role in the agile process.
Michael: Cool, so what lessons have you learned from dealing, working in with multiple teams. You said multiple teams, multiple time zones, multiple countries, multiple programming languages and multiple headaches. I think I added the headaches in, but.
Scott: Yeah, there's definitely headaches and all of that and as we've worked with big teams and whatnot, some have offshore teams in multiple contents even, much less time zones. And so really one of the biggest challenges is communication. And I don't even mean language barriers of people in different countries, but just finding a way to facilitate that communication between all the team members is a big deal. And so we've been super intentional about using some of the more modern tools like Slack and things like that. We've really enjoyed that. That's given us the ability to just really take our communication to the next level beyond things like tickets and email and things like that.
So that's been really, really helpful and in terms of just managing all of the different technologies and things like that, that's where that source control stuff really comes into play. When we determine the source control strategy, we could say, “You know what, this applies no matter what the technology is. This applies for ColdFusion, for PHP, for Node, for C sharp. No matter what, before we learn how the code works we want to learn what the deployment process is. How do I get my code, my code change from my local box all the way to production.”
And when we had this process we could have … We could all be on the same page and have the same language, so to speak to communicate. And that really, really helped us as a big team. And so when we have a release, we have five different teams releasing code in five different languages, and we're able to all kinda do that in lock step and push it all out at once. And so without that kind of a thing, you're really overcompensating on the communication trying to figure out what every other team is doing, and have they pushed yet, and have you pushed yet, and my feature depends on your feature. And we all have to go out at the same time or the whole thing's gonna fall apart.
So that's where it became really, really apparent to me and even more so as we added more teams and more technologies, that we have to have a good strategy in terms of our source control. Otherwise things are just gonna get chaotic and fall apart.
Michael: Mmm, and it sounds like you're more a fan of elegant and simple stuff rather than having fall apart chaos complicated things.
Scott: Absolutely, absolutely. Somebody said the other day the enemy of reliability is complexity. And I just couldn't agree more. And I'm all about the simple solution or the elegant solution. Let's do it simply so that it's reliable and easy to communicate. And we can add complication if we have to, but I'd really, really rather not. So let's be as complicated as we have to be to get the job done, but otherwise let's aim for simplicity and elegance.
Michael: Yeah, I mean to paraphrase, I forget who this quote was from, but it takes more time to write simple code than it does to write longer, more complicated code.
Scott: It does.
Michael: You have to think, you have to think more.
Scott: It does, you have to be a lot more thoughtful about it. And sometimes, having time to be thoughtful is a luxury and not something that we always have, but whenever possible its usually pays off in the long run. Both for the thing that you're writing now and for the person that's coming behind you to maintain that code. I think it's valuable.
Michael: Yeah, it was a quote from Mark Twain saying, “I didn't have time to write a short letter so I wrote a long one instead.” I think the same idea applies to programming.
Michael: So you mentioned that you have multiple time zones and you use Slack. Do you have scrum meetings across the whole multiple teams? Or each team has a separate scrum meeting?
Scott: That's a great question. And so one of the things our teams have been really good about is adjusting work schedules and adjusting meeting times so that everyone can participate. And so we'll have a meeting early in the morning US time and that's late afternoon evening Europe time or India time and super early in the morning West Coast US time.
And so we had to kinda find a sweet spot with the teams that we had and figure out how we can all be on meetings together. And so for a couple of hours every day, everyone is online at the same time and we can exchange notes, we can have meetings, we can communicate as we need to. And then of course with things like Slack or even email, you can … Later in the day I can send something and they're pick it up overnight and that works too.
But everyone on our teams has been really good and really flexible about being available for a certain block of time so that we can all get in the same virtual room for a little while and talk about with needs to be talked about.
Michael: So you mentioned you're using Slack a lot for communication. For folks who aren't using that in their projects. I think a lot of us have seen it in the ColdFusion Slack channel or other Slack channels we have seen, but we may not have actually implemented it on a project. What difference did that make over what you were doing before. What do you like about using Slack on a project?
Scott: What do I like about using Slack? It's to me it feels like that simple elegant solution in terms of communication so that definitely resonates with how I like to think, but it … There's so many creature comforts with it in terms of being able to set, here's my schedule then I'm going to be online and I can get my notifications or not, which really helps with that time zone thing. I don't have to worry about waking somebody up in the middle of the night if I send them a message.
It works really well with going between devices and things like that. So I can be on my phone or I can be on my computer and really the person on the other end can't tell the difference, so that's really helpful for somebody that works remotely and I might be in transit between my coffee shop and my office and my other coffee chop and whatnot.
So it helps keep things a lot more continuous and a lot more graceful. And the channels and everything else that it provides is also really helpful. We use tons and tons of channels, probably more channels than we should. But it really does help organize communication so I can know that if what's being talked about in what channel is relevant to me or not.
I mute some channels because they don't really apply to me but they can still at me and it will send me a notification to let me know that my name was used and I can jump in there. So I've just found it to be a really elegant tool. And we've tried a couple others and it's not as team oriented.
Skype is great, but it's not team oriented and we tried HipChat for a while, 'cause we work with a lot of Atlassian products and we thought HipChat would be good, but honestly we have found ourselves going back to Slack, so it's been a really good tool for us.
Michael: Great, so it's interesting. Do you use any other project tools? You mentioned Slack, but, and you're using Git of course.
Scott: We use Slack definitely as our primary communication outside of email. We use Atlassian products, we use JIRA for our ticket management, their Confluence product for our documentation and wiki type efforts, and we use their Git implementation, which is GitHub. Not GitHub, sorry, SourceTree is the client and Bitbucket is the website. So we use a lot of their tools.
SourceTree I like a lot. I've heard a lot of people talk about some of the other ones. SourceTree's just become very comfortable to me and it works, so I've stuck with it. But I would say those are probably the main tools. I'm looking across my taskbar here to see what else I have [crosstalk 00:23:45].
Beyond Compare is one of my favorite code editing tools, or comparison tools. I use that daily. [crosstalk 00:23:54]
Michael: Now why do you use that as well as using Git, 'cause doesn't Git let you compare what's changed, or?
Scott: Oh I love Beyond Compare side by side comparison, and I love how it lets you set rules for what is important and what isn't. And you can hide unimportant changes. And it's helped me so many times, especially with Git merge issues and things like that, it's found white space line ending differences that I couldn't see anywhere else and I could see it in Beyond Compare. And it helped me figure that out and solve the merge issue.
And so it's not a free tool or anything, but it's absolutely one of my critical tools. I bought the license personally because I wanted it and it's just one of my foundational tools.
Michael: Fabulous. So I'm just curious what you use for editing your code, 'cause a lot of people kinda have strong views on what code editor [crosstalk 00:24:57].
Scott: Right, right. I've used several editors over the years. Right now I'm using Sublime. I enjoy a lot of again the creature comforts it provides, the multi-line editing, the quick selections, some of the different things it provides. It has a lot of nice theming and people have written some nice themes for it. So right now that's kinda my go to editor for really any text editing, including ColdFusion code and that's definitely become my preference.
Michael: Great, well you're with majority of ColdFusion developers at least the ones in our state of the union survey that's the most popular editor right now. So, fabulous.
So, you mentioned you've learned how to avoid deployment day disasters, so maybe can you tell us a bit about deployment day disasters you may have observed? And then what you do to avoid that happening.
Scott: There's a whole lot that goes into that. So many opportunities for disaster when you're deploying. Again, having the code management strategy is probably one of the biggest pieces of it. To have all of your teams on the same page and using the same language when referring to deployments is absolutely critical.
Michael: [crosstalk 00:26:21] There's a different language you can use? What language do you use?
Scott: Well, I use English. But no-
Michael: [crosstalk 00:26:31] Terminology you use, right?
Scott: Absolutely right, so when I talked about merging feature branch. In our world, that means a specific thing. Or running the merge script, or running the this or that. I can say this to my ColdFusion guys or I can say it to a PHP guy or a Node guy, and they all know what we're talking about [inaudible 00:26:54] it's time to run the merge scripts. Okay, we can all run the merge scripts and we're all doing that in lock step to move forward towards deployment.
And so that's what I mean by using the same language. So when we have a process defined, we're gonna call different steps different things. And if all the teams are using that same process, then we can all intercommunicate and know where we are.
And it also provides an opportunity to share some of that responsibility between team members. So it's not that I know all of the secret steps to deploy some code. I can have a colleague or even somebody in a different technology step in and it's all Git management. You don't have to necessarily know the source language. But they can step in and take over that deployment if necessary. So being on the same page, having the same process, and having that same language of what you're doing is a big piece of that.
Michael: What would an example be where someone on the team had a different concept of how deployment works and it kinda messed things up.
Scott: Yeah, so I'll give you a recent example of that is we recently started working with a team that had different language, they kinda were brought in last minute, they had a whole different ecosystem and everything. We didn't touch any of their stuff, so they managed their stuff and we managed our stuff. And so when it came time to even releasing not to production but to a quality assurance level server so that we could verify that the features were doing what they're supposed to, we were running our processes to deploy to that QA environment and we would ask them to do that and they were like, “Oh, well we can't really do it that way. And we'll have to go back and refactor a bunch of things and try to spin up a new environment and we'll have to do all these things.”
And we're like, “Well, why didn't you just do it the way we did it, you know?” And so that just completely threw everything out of sorts. So the QA team is going, “We're ready to test, but the thing isn't up yet and that feature went but this other feature went and we weren't expecting that.” And so we had to kinda massage that and work with them to get onto the same deployment process so that we could all be on that same page and we could deploy in lock step even to a quality environment, much less a production environment. And sort of without that then you do kinda get some chaos.
Michael: Mmm, that makes sense, so that's quite a subtle issue not having a common way to describe how your deployments go.
Scott: It is, it's not the first thing that you think about, but if you begin with the end in mind. As a developer, if I don't care about anything downstream, or upstream, then what do I do with my code change? I got a bug, I got a bug ticket, I fixed some code, but what branch was I on? What do I do with my code? Am I supposed to merge it? Am I supposed to … These are some of the first questions that you ask as the developer, and so having that process to … Well, where is it eventually supposed to end up? Then I can backtrack that and figure out where I'm supposed to operate here just in the development phase.
And so I think it's not the first thing that you think about, because as a developer we think about, “What do I do with my code?” But it is important, again, to begin with the end in mind. And working that out all the way through is important.
And really you don't have to go through too many deployment cycles before you figure out its kind of an important thing because there's a lot of headaches involved if you don't.
Michael: When you said begin with the end in mind, I'm going to imagine you're a Stephen Covey Seven Habits fan?
Scott: I am, I am. I definitely stole that quote, so that's a good one. I like to think about that often and whenever I'm doing something it's like, “What am I doing?” And even in preparing for a podcast interview or preparing for an Into The Box speech, it's like, “What am I trying to communicate here and where am I starting from and where am I hoping to end up?” And so that's definitely a valuable concept there.
Michael: So, any other deployment day disasters that you've seen in your years of coding? You've been doing this for over 11 years now, right?
Scott: Hmm, the biggest disasters have usually come down to special cases or situations where it worked on my machine but it didn't work in production and I kind of had this unique setup locally. And I worked it out there and then once it deployed it didn't work because I didn't expect that setting would be different up there.
And so getting consistency between environments is a huge deal in terms of just having a reliable deployment. Making sure that there's not any special cases or special servers or special anything that you can just reliably and reproducibly deploy your code from my machine to the next machine to the next machine.
If you can get to that point then you avoid a whole lot of potential disasters and those all-nighters trying to figure out what the heck happened.
Michael: Cool, so let's change topics slightly. I've been asking all the people I'm interviewing this. Why are you proud to use ColdFusion?
Scott: ColdFusion is the first web language that I learned. It's near and dear to my heart. As I've seen it evolve and certainly as I've seen it go from a closed source Adobe system to a more open ecosystem, I've just really been proud to work with it. And I love the work that I get to do in it. And I love the work that I get to do it in a modern way, if that makes sense.
A lot of the code that I get to work with right now, and this may be unusual, but we're really doing some modern things and using up to date ColdBox standards with MPC. We're doing a lot of RESTful APIs. We're doing some of these cutting edge things and it really makes it enjoyable and I've been really impressed with how ColdFusion has kept up with that, and how it's even led the way on some of those things, it feels like.
I am proud to use it. People kinda down it sometimes and I, you know what, I just really enjoy writing the code. I enjoy what it can do. And really I feel like it's as performant as anything, so I love it.
Michael: Great. So what would it take to make ColdFusion ever more alive this year?
Scott: Even more alive this year? You know, the developers and the community makes it, so anything we can do to rally more community and rally developers to pull that in for their little pet project or anything like that I think is a huge step. And just getting some of that excitement going again.
So it's mature, it's been around for forever, so it's not the new kid on the block but it's still doing new things. And I love that there's still that innovation both with the language itself and with what Lucy's doing with it. And also some of the para things going on around it with the ColdBox and CommandBox and the CLIs and things like that.
It really is still a very alive and well ecosystem and I think the more we can get people to realize that and to utilize it, I think it's grow.
Michael: That's a great answer. So I'm just gonna go to a [inaudible 00:34:46] and say, you said that one of the things was to get more people involved in ColdFusion? So what would it take to get more people involved?
Scott: You're talking to an introvert here, so I don't know. You know, conferences like this are certainly helpful. Podcasts and things that get the word out there of what's happening in modern CFML. You know it's not CFML of 10 years ago or 15 years ago, it's a whole new world of what CFML is. And it is continuing to evolve.
And so somehow getting that word out there, putting projects together that are easy to work with open source-wise. Things like ContentBox or whole packages you can just download and implement. I think that makes it easier and more lucrative for people to jump in and start playing with this stuff.
Michael: So what you're going to be speaking Into The Box, what are you looking forward to at the conference?
Scott: Some of my favorite people are gonna be at Into The Box. I've had the pleasure of working with Luis and the Ortus team, both professionally and just being around them personally, and so I'm just excited to have that community there and to spend some time with some amazing people.
I always walk away from these sessions inspired to be better, to go learn new things and to embrace the things that I've been exposed to. I walk away thinking there's 32 things I'm doing wrong at work right now and I need to go fix it.
But inevitably there's one or two things that I walk away with that just really, really stick and change the way that I do things on a day-to-day basis, and so I'm excited to see what that's gonna be this year.
Michael: Fabulous, well I'll look forward to seeing you there and thanks so much for coming on the podcast, Scott.
Scott: Absolutely, thanks so much for having me, Michael. This was fun.