Neat! Yeah, that's a pretty complex context and I completely see what you mean about the new hardware being part of the rollout and necessarily meaning that you can't just run both systems. My comment is more of a strategy for just a backend or online processing system change than a physical brick and mortar swap out.

In my note about misreading the suggestion, I was thinking generally. I do believe that there is no reason from a PCI perspective why a given production system cannot process a transaction live and also in a dry mode on a new code path that's being verified, but if the difference isn't just code paths on a device, and instead involves hardware and process changes, your point about needing to deploy a dev box and that being a PCI issue totally makes sense, plus the bit about it being a bad test anyway because of the differences in actions taken or outputs.

The example you gave originally, of shipping to the lower stake exceptional stores first and then working out issues with them before you tried to scale out to everywhere, sounded to me like a very solid approach to mitigating risk while shipping early.

More of the background of the project.

The original register was a custom written C program running in DOS. It was getting harder and harder to find C programmers. The consultancy that had part of the maintenance contract with it was also having that difficulty and between raising the rates and deprioritizing the work items because their senior people (the ones who still knew how to sling C and fit it into computers with 4 MB of memory that you couldn't get replacement parts for anymore) were on other (higher paying) contracts.

So the company I worked at made the decision to switch from that program to a new one. They bought and licensed the full source to a Java POS system (and I've seen the same interface at other big retail companies too) and replace all the hardware in all the stores... ballpark 5000 POS systems.

The professional services consultancy was originally brought in (I recall it being underway when I started at there in 2010). They missed deadlines and updates and I'm sure legal got in there with failure to deliver on contract. I think it was late 2011 that the company pulled the top devs from each team and set us to working on making this ready in all stores by October 2012 (side note: tossing two senior devs from four different teams into a new team results in some challenging personality situations). And that's when we (the devs) flipped the schedule around and instead of March 2013 for the cafeteria and surplus store (because they were the odd ones), we were going to get them in place in March of 2012 so that we could have low risk production environments while we worked out issues (so many race conditions and graphical event issues hanging with old school AWT).

---

... personality clash memory... it was on some point of architecture and code and our voices were getting louder. Bullpen work environment, (a bunch of unsaid backstory here) but the director was in the cube on the other side of the bullpen from us. The director "suggested" that we take our discussion to a meeting room... so we packed up a computer (we needed it to talk about code), all of the POS devices that we needed, put it on a cart, pushed the cart down the hall into a free conference room (there were two conference rooms on that floor - no, this wasn't a building designed for development teams) and set up and went back to loudly discussing. However, we didn't schedule or reserve the room... and the director that kicked us out of the bullpen had reserved the room that we had been kicked into shortly after we got there. "We're still discussing the topic, that will probably be another 5-10 minutes from now... and it will take us another 5 minutes pack the computer back up and take it back to the bullpen. Your cube with extra chairs in it should be available for your meeting and it's quiet there now without our discussions going on."