Automation Implementation: When the Robots Are Ready But The Warehouse Isn’t

[00:00:00.000] Howdy, Mark.

[00:00:01.240] How’s it going, Charles?

[00:00:02.820] It’s the end of the week, so how can I not be happy?

[00:00:06.600] Very true. End of the week is always fun.

[00:00:08.960] Do you have good plans for this weekend?

[00:00:10.990] I actually don’t. This week is a recovery weekend for me. I did a little trip down south last weekend, so I’m taking it easy.

[00:00:17.820] Where did you go?

[00:00:18.770] I went to St. Martin.

[00:00:21.020] Oh, nice. Was it fun?

[00:00:22.800] It was a great time. Just me and a couple of friends. We decided to do a weekend in the south.

[00:00:27.880] And the weather was good?

[00:00:29.010] We had a little bit of rain day one, but then it cleared up. Enough to burn my fair skin.

[00:00:35.860] Yeah. Well, that’s great. But really, we have business to discuss right now, don’t we?

[00:00:42.940] Yes, we do. And The world of automation.

[00:00:46.930] The world of automation, but from a different angle, which is just an interesting story that I thought it would be great for you to share with everyone. So let’s set the scene. This is not a story of somebody bringing us in to help them automate a facility in the classic ideal way. Tell me How did that happen, Mark?

[00:01:16.130] Yeah, of course. So one of our customers, actually one of our business partners, was supporting a customer and putting a new facility together that had a fair level of automation. Throughout that project, their Their subject matter expert that they had internally, I believe got promoted. I’m going to go with that instead of got let go, but I think got promoted and moved away from the project. Therefore, they had nobody left on site to support with the iddy-gritty, the day-to-day of putting this thing together during the install and making sure that all parties remained accountable. So they reached out to Lid asking for support in having somebody on site It was a 14-week on site overall to help them, one, make sure that the installation was done correctly, but then also all of the integration and testing.

[00:02:12.760] So in other words, 14 weeks at the end of the show, when a project like that is a 24-month type project means that you’re coming in in the very last act And all the sins, omissions, and warts of the thing are for you to manage at the very end.

[00:02:39.630] Yeah, exactly. I think there’s the notion of coming into chaos. It really is It really was that. It was a lot of active parties trying to close things out, but also wanted to make sure that things were done properly and that you were meeting all as per contract specifications. And it definitely was a little bit of a rapid learning curve to make sure that we can add value quickly to the project. Some of it was just making sure, like I said, the install went correctly. But as soon as you get into that testing phase and all of a sudden things aren’t working properly, you need to make sure you have that 500-paste contract memorized and be able to circle back to it and call out things that don’t make sense. The goal here was also to be an owner’s rep for the customer, for the client, to make sure that the automation provider didn’t just cut any corners and install as they saw fit, but really met the customer requirements.

[00:03:42.480] You split this into two work streams. It’s just physically get it all in, coordinate multiple different suppliers of installation services and of equipment, get it all in. And then the second stream is to represent the owner when testing comes along.

[00:04:08.630] Yeah, exactly. Those are really the two phases that we were supporting with.

[00:04:13.050] And you had inherited a testing plan?

[00:04:17.230] We inherited one. We added some notes and tried to give them some red flags to watch out for. But at the end of the day, we did inherit one. We worked a little bit with the customer to fine tune some of the components. Honestly, the big back and forth that happened was what is considered SAT, so site acceptance test, versus what is considered user acceptance test. How do you How do you accept something that you’re not sure can do all of your non- happy path elements? And I think that was a piece that was a lot of back and forth between all the parties to make sure that we’re all aligned.

[00:04:55.580] Just help everyone. You’re already out of the gate. You’re making this distinction between Site acceptance testing and user acceptance testing. And just help everybody who isn’t familiar with that. What would you, by default, say is the difference between the two?

[00:05:12.110] Okay, perfect. So a site acceptance test means that the system is still owned by the automation provider, and they are performing their test to ensure it meets speeds and throughputs that have been defined in the contract. The user accept this test is when It gets signed off to the customer, and they can do whatever they want with the system to test it when it’s not necessarily going through the ideal paths to make sure it can handle all of the bad labeling or miss palettes or heavy palettes and all of these different little points that are more unique to their operation that they want to make sure the system can do that is outside of the scope of the contract itself.

[00:05:59.000] So in other words, I’ll just restate it so it’s clear. Site acceptance testing is something that the, call it the solution provider does as an internal check for themselves. Not that they’re hiding it from people, but it really is. It’s about reviewing end-to-end the solution to make sure that when they hand it over and say, Look, here it is, they have to their best delivered what they promised they would deliver, right?

[00:06:31.480] Yes, exactly.

[00:06:33.090] And then the user acceptance testing will probably do some validation of that site acceptance testing. And then they’re going to, as you said, they’re going to run through a series of scenarios where things go wrong. And part of that is to see, okay, when things go wrong, here’s how the system reacts, and let us develop our contingency processes because things are going to go wrong. The label example is a good one. And then the other part is, well, is it performing how we expected it to? And if it doesn’t, then there’s a conversation that you have between the client and the vendor to iron out, is that a fix? Is that a fix that I pay for in addition to what I’ve already paid for? Is this an unfixable? And at that point, as the kids say, WTF. Yeah.

[00:07:38.670] Is it really a true limitation to the system that was not planned for in advance? Or is it something that we can work through. A lot of it also is going to be making sure the system reacts the right way. Obviously, if you have one piece of automation, it’ll run through it. But if you have conveyors leaning into different robot arms and different aisles of automated system, you want to make sure that the logic within the system also makes sense. So understanding that that back-end would be was also a big piece there. And that’s also where the system ties in and UAT, where you’re getting live feeds from the host versus only getting prompted feeds that mimic a host during SAT. So there’s that notion of making sure that the system is properly integrated and that you’re acting upon the right notice and the right triggers versus just- Just to- Yeah. Sorry. Keep going. I was going to say versus just manually creating those triggers. Okay.

[00:08:39.540] I was going to say, are you making a distinction here between… So we know that there is In any automated system, there is a mechanical component, and then there’s a technology component, software. And ideally, the software, both the integrations and then its performance is being tested even before the equipment is being bolted into the warehouse floor. That is a different set of testing that is occurring concurrent with just the construction of the warehouse and the building out of the install. And that’s where I’m not really running with live data. I’m mimicking what live data would do, right?

[00:09:28.890] Yeah, exactly. You’re mimicking It’s only until you get to your user acceptance test. In that case, you’re able to plug in because you’ve not taken more partial ownership or temporary… What was it? Not temporary. I’m not listening to the word of what it was, but it’s a temporary or an initial step of ownership without the final sign off. And that’s where you get to plug in and really see if those prompts truly work. Yes, they’ve been worked on in the background, but you’ve never fully seen them in action.

[00:09:57.120] A hundred %. Okay, so now my question is, how How long was… Let’s forget the site acceptance testing. I think everyone understands that the user acceptance testing is really from the operation perspective, the most important thing because this is your last chance before you start shipping cases to really get kinks fixed. So my question is, how long was the user acceptance testing period?

[00:10:26.340] So this one was a little unique. When we first got into the project, there was no time defined for user acceptance test. This was a red flag. We got that right away.

[00:10:34.830] How did that…

[00:10:35.840] It was not- It was not in the contract. It was not in the plan schedule. It was not defined as one of the tasks or one of the sections that need to be completed before getting final sign off.

[00:10:48.050] Which is something you would never recommend that approach, would you?

[00:10:53.970] No, not one. I mean, you need to be able to take ownership of the system, start dealing with it, understanding it a little bit, and putting all of your unique components and unique processes through it because you’re not going to know if it’s truly doing what you needed to do. So as soon as we heard that that was not… As soon as you got the schedule at the beginning of this mandate, We rose a flag right away saying this needs to happen. When can we do it? A red flag. A red flag. And the other notion was doing it at temperature versus not doing it at a temperature, right?

[00:11:27.550] This was a temperature-controlled building.

[00:11:29.650] This was a temperature-controlled building. We had freezer products. We were working in a cooler environment for a lot of the automation components. Making sure that the automation can handle that level of moisture within the different items moving through it was a pretty big deal. This is a lot of discussions and a lot of pushback at exactly when we can do this because cooling down the building was planned for specific times, did not match up with when they wanted to perform UAT. So there is that whole notion of coordination and planning and having all this in a schedule is truly important because there’s so many more aspects of it saying, Hey, I can turn refrigeration on. You’re dealing with a system that’s 100 feet high. Cooling that system down does not take a week.

[00:12:20.980] Twelve hours. It’s a huge undertaking.

[00:12:23.490] It takes a long time because you’re cooling it piece by piece. Day by day, you’re dropping temperature. You also want to make sure that a system maintains temperature. You’re going to find the leaks in the cladding. You’re going to find a lot of different components. So getting that temperature down is going to be really important. And also before you get to your UAT, you don’t get to test with real product. So they’re doing all of their testing with sample product, which tries to mimic it as much as possible.

[00:12:50.310] Hold on, hold on. Sorry, sorry. Just not to be too nitpicky. That’s okay. They are doing all their testing. Now you’re talking about site acceptance testing.

[00:12:59.480] Yes. Sorry.

[00:13:00.410] Site acceptance. The vendor site acceptance testing.

[00:13:01.750] The vendor is doing… And this is also because the customer did not want to provide real product and then waste that product. So there’s a notion here of putting a budget towards making sure you can scrap some of your real product because it’s the only thing that will give you real result as you’re looking through it. A case won’t pass through the same unit 10, 15 times. That box integrity will not withstand 10 times running through it, which is what’s happening here with potential fake product. So making sure you give that time and you bring things down to temperature is going to make a massive difference to understand, one, can it really do what it needs to be doing? And can it handle that moisture, that level of moisture?

[00:13:47.400] Okay, but let’s step back. I mean, that’s great stuff, but step back for a second for me. So you see the schedule. There is actually no user acceptance testing in the schedule. You raise the red flag. How do people receive that? Because I can imagine there are a bunch of panicked people who are trying to deliver a date to folks at headquarters, potentially hundreds of miles away, remote, and just get not on the ground. So how does it get received when you say this could be a disaster if you don’t properly test the system?

[00:14:30.560] There’s a few different ways. Obviously, everyone’s trying to bend over backwards to accept everybody else, but they’re also trying to do that without hindering themselves and what they need to do. So as soon as we find out that we have a couple of days to spare in some site acceptance test, that we were able to do things a little bit quicker, we’re trying to go get some days back. At least the automation provider did handle and had some buffer embedded into the schedule just slightly. Obviously, it was more in case that they had some issues. So we were able to bounce that around. It was a lot of conversations. It was a lot of, call it executives, putting their foot in to say, hey, this is almost a non-negotiable here. If we can’t go in there, we’re going to delay the go-live. So they were willing to push things back to make sure that they got what they needed, which I think was essential. But these are not decisions that are necessarily being made on the floor, to your point. These are going back to headquarters. And then you have leaders in both both companies coming together and making this happen.

[00:15:33.950] These are not small projects. So there’s a lot of money on the line. Of course not.

[00:15:37.720] Absolutely.

[00:15:39.140] You want to keep that relationship. You want to keep that rapport. But it wasn’t easy conversations, and it wasn’t conversations that were fixed within a day or two. This did take 2-3 weeks to just iron out and make sure we can find that time in the schedule, as well as talking to the construction company, asking them to cool things down earlier. It was all intertwined there.

[00:16:02.660] So lesson one, though, is no one should get into a $30 to $50 to $100 million investment in automated material handling without some user acceptance testing at the end of these things.

[00:16:18.100] No one should not have user acceptance tests, especially with these larger systems.

[00:16:23.170] If you put it in the beginning of the project schedule, you will not have people wringing their hands when that comes long.

[00:16:30.520] Exactly. Exactly. Making sure you’re planning. Plan it for it. Plan for it ahead of time because you won’t get the pushback that we got for sure.

[00:16:39.170] So you had roughly how many days of user acceptance testing?

[00:16:44.440] There was We ran, I believe, for two weeks, so 10 days worth of testing.

[00:16:49.060] And how many scenarios had you designed or defined, however you want to say it, Ahead of that user acceptance testing?

[00:17:03.130] We must have had, I mean, 40, probably different scenarios that they were trying to run through. I think there was a limitation on time where we just we accepted some regardless. And you try to overlap them where you can do a test to start it, but let the product run through because you can hit a bunch of different tests along the way. So it’s about resetting it continuously and making sure that you can try to sequence five or 10 of them in a row. If they had their way, they would have probably done three, 400 tests. They would have just kept running the system. There is a notion of reliability in a system as well, making sure that the uptime matches what you expect from the contract, which was another point that I hadn’t seen in the contract, which was a bit of a concern, not having that reliability and availability of system to say that you need to be active or uptime for 98. 5 % of working week. So a lot of little concerns to make sure that we can validate the system. The red flag that came up early.

[00:18:10.360] Yeah, that’s fascinating. So now, explain the process for people. You do an unhappy path test and the system fails. Now, what you could do is start panicking and screaming and yelling. Probably not the most productive way to go about it. How did you undertake… The system fails, there’s a diagnostic that happens. The client and the vendor are collaborating in this diagnosis, correct? Yes.

[00:18:49.110] So what happens is, first off, as soon as something fails, we want to identify why it failed. If you’re not able to identify why, then you might want to run it again because it might just been a fluke. So there’s always a notion of, hey, let’s give it one or two, maybe three extra times to see, is it doing the same thing all the time? Is it a fluke of what’s happening? And then what are the reasons every time that you’re coming through it? Once you’ve defined all of the reasons, it’s been three or four times, the same reason comes back over and over, you’re actually going to just take that out. You’re going to mark it down, and you’re going to put it on this checklist that needs to be checked after. I don’t want to get lost in this initial to ask because I have a lot of tests to get through. I want to pull this off the line, get IT of both ends involved because understanding where that originates from, and then coming up with a potential solution to run it virtually because you can virtually run it through the system as well and see how it would go.

[00:19:49.650] So there’s always that notion of let’s not waste time on the floor figuring out all of the details. Let’s put it aside and move on.

[00:19:57.880] We weren’t- Put it in a parking as the cool kids say.

[00:20:01.760] Exactly. And we weren’t involved in the fixes. We were really the eyes on the floor, identifying, recording, making sure that things happened as per contract. So I never got to see the whole continuation in this case of what that would look like. But they did do the right thing of putting in the parking lot and continuing on with the.

[00:20:23.570] Yeah, that makes sense. Right. You’re going to make… Because you don’t want to get dragged in a… It’s But actually, you know what? It’s like doing a final exam. You got 13 questions. And if question two stumps you, don’t let that stumping prevent you from answering the other questions. So you skip question two, and then you finish the others, and you go back to question two. And hopefully, you’ll have some clarity, or at least you didn’t lose a whole bunch of time, a precious time. All right, so you go through. And then, of course, let’s say you diagnose it, you know what the fix is. You’re going to make a decision about the criticality of the problem and the magnitude of the fix, just like anything. You’re going to create a punch list. You’ll decide what needs to be fixed right away, what can be fixed post go live, because we can live with that, but we need to fix it. I mean, I’m assuming, of course, that that’s how- Yeah, they had a punch list.

[00:21:26.880] So we had a punch list with a priority on it as we’re running through it. It’s like to your point, some of them are not critical. It’s moving a couple of stoppers to make sure that pallets slow down and stop in the right areas for the system to operate. And there was others that were, Hey, this is critical. We cannot go live without it. And once you create that punch list, you want to review it on both ends. Obviously, here the automation provider has a bigger pull than the client, depending on the solution. But, hey, when can you have this fixed by? So also creating timelines and adding dates to all of those fixes to then come back and retest it once that fix has been complete. Right.

[00:22:07.010] All right. That’s good. So what would you say are some of the key takeaways? I think what’s really interesting because a lot of people… We get a huge audience now, Mark. I don’t know if you know this, but it’s enormous. And a lot of people are always curious about what goes on behind the scenes. It’s pretty exciting for a lot of people. And one of the things is, sometimes you walk into a conversation thinking you’re going to talk about one thing, and then you realize you’re actually talking about something else. I do think this is a really good example where there’s so much to think about in terms of setting up a good user acceptance testing program. Folks in IT are much more versed in this than operations people who, in the old days when you’re putting up racking and buying some reach trucks and some pallet jacks, there’s no such thing as user acceptance testing. So this is a brave new world for a lot of people. And I think there’s a lot of value in the discussion we’ve just had. Just not even talking about anything more than just what user acceptance testing is.

[00:23:31.340] And so what I’d like to know, we mentioned a few key takeaways, obvious one. Have it scheduled from the get-go, right? Because clearly… Anyway, I won’t even belabor that point. So what are some of the other key takeaways that you would want everyone to know?

[00:23:56.430] Some of it’s going to be just having the right people on site, understanding who’s going to go through this process and having some of your engineers and hiring staff that will be working on the system. I think that’s pretty critical. It’s one of those things where people put in automation to reduce labor count, but the labor actually shifts. And if you’re doing all of this acceptance, you have expertise on site that’s doing all these fixes. You have all the people from the automation provider helping through. Sometimes it’s as simple as a chain that’s just a little bit too loose or It could be as simple as rollers that don’t work or any little component like that. Having your manpower on the floor during acceptance test that can understand and try to diagnose it at first will give them that expertise for the future. That’s definitely one of the components. Second one is run your test by your provider. Whoever provides it might tell you, Hey, don’t even run this test. It’s impossible with the system that you have. You want to make sure you don’t waste time as running through it and have your staff on site for site acceptance test, which is not your user, but you’re still looking at the system.

[00:25:08.500] They still allow you to hover behind and take notes and understand what’s going on. I think There’s a lot to run through there, and there’s a lot to grasp as you’re watching them work that might help you or might allow you to find other tests you want to perform because, Hey, this one was struggling a few times. Yes, they passed, but we got to run this one a few more times because we don’t trust it. So there’s a lot of just… It’s a trust. Obviously, it’s a trust thing between you and the provider. There’s a big back and forth, the sign offs, but then there’s creating an extensive list and time it takes to run through those tests and make sure that all makes sense.

[00:25:53.220] I love it. Well, I don’t know if we need to dwell any more on this, because I think Just the concept of taking user acceptance testing very, very seriously is really valuable for a lot of people who would otherwise stumble into things, having spent the money and now left with maybe a mess. The tragedy of the mess is it’s a punch list that can be fixed. Do you know what I mean? But the move The way people would react in the absence of that proper testing, if they just went live and had a crisis on their hands, it’s a hostile atmosphere that is unnecessary.

[00:26:52.680] You know what I mean? I do. And it’s one of the things that you want to come back and think like, the service provider wouldn’t give me a system that doesn’t work. Their name is on it. So you want to trust that the site acceptance test is going to get the job done. But again, you don’t get that integrated platform. You don’t get the real live communications. You don’t get a lot of the pieces that are going to be just critical when you do go live. To your point, all go live is a technology. It’s a bit of a roller coaster. You’re going to get some hiccups. It doesn’t change here. It’s going to be the same thing. So the more kinks you can get out early on, the better.

[00:27:29.450] Absolutely. Well, Mark, on that note, I think we will say goodbye to each other. I hope you have a great weekend in Atlanta and hope to get down there soon and enjoy that wonderful city with you.

[00:27:44.180] Amazing. Well, it was nice talking to you again, Charles.