September 1, 2016

RBS (First Time) - Part 2

Tags: ,

As the SCRS project at RBS began to come to a finish, the team started to disperse. I was the last technical person to leave and the SCRS project manger arranged my contract to transfer to the Sales Account Opening (SAO) project so I could still be around if SCRS needed me. They didn’t, so my last 9 months at RBS were spent on the largest most dysfunctional IT project I have ever seen.

SAO aimed to allow potential and existing customers of RBS to open accounts online. Current accounts, savings, credit cards, overdrafts, the whole range of retail account should be available. Basically it was a set of webpages that took a customer’s information, and then passed it through to various backend systems to actually create the account and send out any required paper forms. At the time many banks were setting up similar systems – now it is expected functionality for a bank’s online presence, but still fairly new for most banks in early 2004 when I started on the project. The technical difficulty was not particularly high, but greatly complicated by non-technical factors. The final system had to integrate with the existing customer database and credit checking systems. The information collected had to meet anti-money laundering, fraud and terrorism laws (collectively referred to as “Know Your Customer” or KYC). Security and reliability had to be bulletproof as this was a public website and had to inspire confidence in potential customers. The site also needed to look pretty, well-designed and enticing. The webpages needed to conform to UK & EU disability laws, so blind people or people unable to use a mouse or a raft of other issues should still be able to use the system easily. Also, RBS owned a number of subsidiary banks. They would all use this system too since they all shared the same backend. This meant the webpages had to change branding, wording and structure to match the bank being viewed.

From my point of view one of the biggest problems was management. The project was huge. At one point I counted over a 100 technical people permanently assigned to the project, plus managers on top. Then there were consultants in various forms: web designers; security experts; usability experts; QA; and more. All this was across three main locations: Edinburgh, London and Bangalore. There were also small (under 9 people) teams in Dublin and Leicester. The budget seemed endless and it was super high profile. We were told the project was initiated by the CEO’s and he was taking a personal interest in the process. It was kind of a big deal. With something this big and important it all became unmanageable and unruly quite quickly.

I joined the project just as development work began. My position in London was officially Java Team Leader. I managed a team of 4/5 programmers (two of which came from SCRS) to create an example system of the webpages. The idea was this would form a template for the mass of coders in Bangalore to follow in creating the real pages. We had to work out a way to transition between pages based on what information was needed next, how to integrate with the interfaces provided for various backend systems and how to have the site rearrange itself depending on what bank was being viewed. There were three other teams in London: the team creating the external interfaces to the backbend databases; the page design team working out what needed to be on each page; and the QA team. We all sat near each other.

My team’s task was completed after a few months and we didn’t have much to do. We helped deal with any issues arising from the work being sent to India, created tools including a comprehensive, but complex build system, and worked with the developers at the printing office in Leicester to get the printing of letters automated. I asked to be released from my contract after 6 months on the project, but my direct manager refused. Normally this would not be a problem, I could just move on with a month’s notice. However, my hopeless agent (who I never worked with again) created a fixed term contract, rather than the standard one month’s notice, so I had to wait an extra 3 months. The manager wanted me and my team around as contingency in case problems arose. Such problems regularly did occur, but never enough to keep us fully occupied. I read a lot of web magazines.

I also helped out lots of other teams. In particular I worked with the Indians temporarily co-located with us, assisting them understanding what we had done and why. These guys were consultants from one of the large Indian outsourcing firms. It worked just like my experiences with other consulting firms. Smart and talented people were sent to sit with us in our office. They gave us confidence everything would be fine as any of them could be part of our team. Then they went back to their main office and most of the actual work was done by much worse programmers. Nice to know consulting firms are the same worldwide. Although one of them did provide one of the funnier work moments. A young Indian consultant was saying how great it was to travel around from his London base. He was particularly impressed by Egypt and asked a quieter member of my team if he had been there. This guy (previously from SCRS) was an Israeli man nearing retirement. He replied that he had been to Egypt, in 1973, and did not enjoy himself. Oblivious, the consultant carried on espousing the country’s virtues.

Towards the end of my time at RBS the Indian consultants in Bangalore sent us photos of their office. We sent back some of hours. It was illuminating. Their offices were modern, open spaces, with large desks and decent equipment (flat screens, etc). Ours were run down, crumbly, ancient offices near Aldgate East, scheduled for demolition (as they have been – the site is now apartments). The offices were a set of 50’s buildings threaded together into a warren of weirdly shaped spaces. We used CRT monitors (the last place I worked with these) and all our PCs were at least a few years old. Though the place did have a decent canteen. Apparently it even used to have a bank branch and pub in the basement, but these were shut down a couple of years before I arrived.

As the end of my contract approached the project began to head off the rails. A specialist project manager was hired – his experience was in bring out-of-control projects under control. Time and budget allocations were being greatly exceeded. Part of this was just the size of the project and how hard it was to get everyone working together. Partly it was due to mid-project changes. I remember being part of one impromptu meeting were the latest changes were being discussed. It seemed that the CEO had seen our latest iteration and wanted a few modifications. Unfortunately, the changes meant the site would not longer meet various legal requirements. A concise paraphrasing of the conversation is “the CEO wants this change. That is illegal. But the CEO wants it. It is illegal!” I left the management team to untangle that Gordian knot. At least that meeting was informal so I could just leave. Most were not, and there was at least one hour long meeting per day. I started taking food to them and eating at the table if it went over an hour. This had the benefit of keeping me alert and the other participants tended to become more focussed on finishing too.

When my contract expired, I left with no regrets. The RBS IT organisation was too bureaucratic for me. I didn’t even have another contract lined up. I didn’t care, I’d had enough of meetings and forms and arguing about budgets. At this stage I hadn’t yet completely given up on becoming a manager, but it was getting close. About a year later the RBS account opening system was available online. I took a look and saw some of my work evident (at least in the way the pages flowed and the html suggested my branched branding was there too, but I couldn’t be sure). There was also quite a bit missing. The project came in about 6 months late by lowering the scope of work. I assume the rest was added in over later months, but I never looked again.

Some other points about my first time at RBS:

  • As I was leaving, I interviewed at RBS Markets, their investment banking division. They seemed to have a very low opinion of their corporate and retail brethren.
  • While waiting in the office lobby for that interview, a set of security people can through and cleared everyone, except me sitting in a corner. Then the bank CEO and Prince Charles walked through. I felt like having a few words with the CEO about how his desired changes to SAO were illegal, but stayed still.
  • HR seemed aware there was a morale problem. Although, instead of making conditions better they regularly ran “fun” and “quirky” campaigns. Putting up colourful motivational banners across the entrance, or handing out buttons that said “willing to play” to employees.
  • Nearly all the non-management, non-Java people wanted to learn Java. I was constantly asked by database or COBOL people to help facilitate them into a Java job on my project, despite them having no experience.
  • IBM salespeople seem to be able to hypnotise corporate IT managers into signing silly contracts at will. I saw evidence of that back in Australia, at RBS, and the next job too.
  • The standard stack on my projects was a Java Struts frontend hosted on IBM WebSphere. Integration with backend mainframes was through a bespoke web services system using Axis. Any local (non-mainframe) data was on DB2. Ant, JUnit and CruiseControl were common. Design was done in the Rational tool-set, including source control in ClearCase.
  • A team member said later that a particular moment made a big difference to their career. We needed to write a web service to interface with the printers. I gave the work to that guy, and he was worried as it was his first big programming task. He was asking me lots of questions about how it should be designed. I told him, just do what you think is best, and if its no good, you will do it again and better. He told me that my confidence in his ability to do the job was a huge boost (as he was not so confident), and it was well-placed because he did it just fine the first time.

August 31, 2016

RBS (First Time) - Part 1

Tags: ,

After Java developers became surplus to requirements at the MCPS-PRS Alliance I had to suddenly hunt around for work again. The PRS people were very nice to me and had no problems with me looking for new jobs while at work. With almost year of working in London behind me, it was surprisingly easy to get interviews. Rather than the months of waiting previously experienced, a number of positions came through agents quite quickly. The best of these was at Royal Bank of Scotland (RBS), in their corporate and retail banking operations (very separate and distinct from their investment banking operations).

So I began a 3 month contract as a Team Leader on the Service Charge Review System (SCRS) just as it started. The project had a 12 month timeline, so if things went well I expected to hang around for further contracts. SCRS was driven by the corporate relationship managers. These are the people who dealt with businesses, big and small. Unlike retail customers (normal people going to bank branches), businesses and corporations paid a raft of fees for the services they use (thus service charges). While there are a set of “standard” fees that businesses pay by default, in reality all the fees are negotiable. The bigger the business, the more negotiable the fees they pay. There are a large number of possible fees, from receiving a cheque, running a credit card or having an overdraft. Perhaps a medium-sized consumer firm has a need for large amount of cash-handling (cash passed to a branch has to be secured, counted and transported), but rarely makes international transfers. It might look for the cash fees to be lowered in return for higher international fees. Part of a relationship managers’ job is to work out if that is a change the bank is willing to make, and if so arrange it. The relationship managers also create speculative fee structures to tempt business to shift banks.

At the time I joined RBS in May 2003, this process was performed on a spreadsheet in an ad-hoc process. Creating a moderately complex fee structure could take over a day and was both error-prone and hard to modify later. This was seen as a bottleneck for expansion. The plan was to create a simple internal website that could import data on current fee structures/usage/profit, allow the quick creation of fee scenarios (often from common base templates), and then save them centrally for future use as templates or new relationship managers (as at the time, people leaving meant all their scenarios were lost). The goal was that a half dozen scenarios for a moderately complex business could be created in a morning, and then easily adjusted based on customer interaction. Based on my consulting experience I didn’t think this was a particularly ambitious objective. It was a matter of loading data, performing some simple calculations, displaying the results on a screen and allowing editing, then saving it again afterwards. There would only be a few hundred users, each of would be trained to use the system, and no one expected it to look pretty. A reasonably straightforward job.

The first task was to recruit the programming team which I would manage. There was the project manager (who hired me), an analyst, a database person and a junior Java developer already on board. However there was budget for two more Java programmers, but they had to be internal hires. At that time RBS was just in the process of shifting from using mainly COBOL to Java for its internal development. Hence all the other developers had only a couple of years of Java work total amongst them, and I was expected to mentor them through the project. Internal hires also worked on the basis that HR would send me a single CV for each position, and I could reject it and ask for another, but it was frowned upon. The project manager strongly suggested that I accept whomever was sent unless there was a serious issue. In the end it didn’t matter. Both the suggested developers were fine.

Along with the traditionally COBOL environment came a waterfall process. Luckily, my manager was an up manager and was not greatly concerned about how the project was actually developed as long as he knew what was happening and it all progressed nicely. So SCRS was developed iteratively inside our small dev team, before being handed off (in a waterfall-like manner) to the separate deployment and support teams. It also helped that SCRS was a small project among giants in the corporate IT section. A couple of large 50+ developer projects (one on a unified document system, the other I don’t remember) had started around the same time as SCRS and were sucking up developers and managerial attention. We were largely ignored and left to our own devices. We started fortnightly iterations and developed a prototype which got a couple of the relationship managers involved on a part-time basis giving feedback. They were generally happy we were doing as they requested. At the end of each iteration we had a working (but incomplete) system that could be checked. Progress was steady, my team was capable and learning fast enough. Confidence was high. Everything was fine.

RBS was the start of my contracting career in London. From that point until now, I have not been a permanent employee. IT contracting may have a bad reputation in some countries, but not in London. It is common among more experienced developers who do not wish to become managers. There is a large community of contractors, many support services, pay rates are high and jobs plentiful. It is assumed that in downturns, contractors are dismissed first, but in my experience pay rates just decrease instead. I started as a contractor by chance; the RBS job was specifically a contract, so that is what I did. My agent suggested an accountant and off I went. Unfortunately, my inexperience caused problems. The accounting firm was borderline incompetent. I sacked them after a couple of years when I learnt enough to do their job. They also suggested a financial advisor who only promoted schemes that involved huge kickbacks to himself. Hmmm, I managed to avoid all of them. Also the agent set up the RBS contract in such a way I got caught by IR35, and stayed that way for the next 5 years. IR35 is the UK tax law that determines whether a contractor is considered a “disguised” employee. Most contracts are written to avoid falling foul of this law, saving the contractor 5-10% on their tax bill. Market contract pay rates take this into account.

While at RBS, I was still considering becoming a manager. So I let it be known I would consider going permanent if the circumstances were right. As a result I got pulled into an HR meeting to discuss becoming a permanent employee. However, I left determined to stay a contractor – at least while at RBS. They explained to me the career progression and where I would fit in. Basically, they expected me to take a step down from my current team leadership position to become a straight developer, although I would still have the same duties temporarily. I would also earn about 40% less even after all the little benefits were added in. Plus I would become subject to all the normal HR forms and reviews. It seemed like a derisory offer. So I stayed a contractor and never looked back. The longer I stayed at RBS the clearer the correctness of this decision became. Just being able to avoid the bureaucracy of a large company turned out to be a surprising benefit and something of which my teammates became immensely jealous. Also there was little job security at RBS for the permanents. There was a common greeting among RBS IT people, “how many purges have you survived?”. Every few years a large number of people were made redundant. Morale was not high.

RBS corporate and retail IT was (and probably still is) an incredibly bureaucratic organisation. There were so many rules and forms to fill in for any occasion. Budget was sacrosanct. Here are a couple of stories to illustrate this. When I first arrived I didn’t have a computer setup properly and I didn’t have the permissions to do it myself. Luckily there was a sysadmin just one row over setting up about 100 PCs just like mine for another project that would be turning up a week later. Each took a few minutes, so I asked if he could do mine and he replied no. It required a form and budget. For a few minutes work? Doing the form would mean waiting a week that RBS would be paying me to do nothing. But still he refused. So I waited until he went to lunch, and added my PC to his pile. The next day I went and collected my PC from the pile and got to work.

Another time I was discussing how much database space we required on SCRS with a db admin (not the db developer). We estimated just a couple of GBs and the admin told us we could have it in a few months, after the budget was confirmed. What, why so long? I offered to give him a few pounds from my pocket as the cost of that much space at a high-street computer store. He laughed and told me how much it would cost at RBS. Many hundreds of pounds, per GB, per year. Huh? It turns out the cost was for the overstaffed (as he admitted) db admin team to justify their existence by forcing captive clients (we couldn’t get the space elsewhere) to pay their budget.

About 6 months in, we hit a couple of problems. This point was actually near the end of the development process. Creating the system was scheduled to take 9 months, with three months deployment, handover, training and support from the dev team, before our part was complete. The first issue was us Java developers racing ahead of the database developer interfacing with the backend systems. We were constantly told that it would be done soon. We agreed interfaces and wrote a mock database so we could continue work. However, the lack of the actual system and associated integration risks were now the top issue on weekly status reports. It was always coming “soon”. Then the database coder just disappeared for a few days. Then they returned and were clearly drunk at their desk every morning. In retrospect, at this point we should have cut them and rushed to get someone else. However, there was some sympathy for them (there were a number of known personal issues), so it was referred to HR and we tried to keep going. Involving HR had no noticeable effect. A few weeks later there was still no database backend, the dev was still drunk at their desk every day, and the rest of the team became blocked waiting on them. Now they had to go. We got an emergency replacement. The new database developer said there was nearly nothing to show for months work, but got to work and pulled some overtime to get everything up and running in just a couple of weeks. To our benefit, by this stage we Java programmers knew exactly what we wanted from the database and had the time to help the new guy get it all running perfectly for us. Although this process lost us about a month off the critical path on our schedule. Still we had built up a little slack beforehand, plus the 3 month slowdown to finish the project could include more development. So it was ok, final handover could still occur on time.

Soon after integrating the database the second problem hit. We presented the a near complete version of SCRS to our customers. They were happy, as they had seen it before on their weekly visits to our office. They took it away to try it in a production setting and show some other relationship managers. A few days later they came back and stated that there was a problem. They need it to be greatly changed. The changes were significant, but not difficult, just time consuming. A month of work at least, probably two. Considering the time lost with the database, there was no way we could make the changes and complete on schedule. More importantly from the RBS management process, our budget would be bust. We asked “why do you need these changes, you saw it being written and were happy with it?” They said we had absolutely delivered what they asked for and wanted. However, over the last few days they realised it was not what they needed. Urgh. They played hardball and threatened to refuse sign off, apparently this was a big deal in the RBS IT organisation, and would reflect very badly on the development team. Our manager countered that we had no money to continue.

After a couple of days an agreement was reached. The relationship managers accepted that the requested changes were new work and not the result of any mistake on our end. Thus they agreed to pay the budget overrun out of their own money. Although not all of us. The dev team would shrink to me and one other Java person for 3 months and one other programmer for half that time. Everyone else would move to other projects. Our manager also got the two relationship managers acting as our customers to sit with us full-time (they would still do their normal work, but from our offices and ready to answer our questions as required). This last point was immensely useful, and probably the reason everything got done to the new schedule. It also meant that news of the project began to spread among both the IT department and the corporate banking department. The relationship managers were very happy with the work being done and the progress on their “extra requirements”. As friendly and talkative people they started telling everyone around them about it. It was like having a marketing department for an internal project.

It didn’t take long before we had early builds going out again to the test users, and the responses were extremely positive (much more so than the first build). It seemed the extra requests really did make a big difference. Thus as the original 12 months expired and we started our handover process, something strange occurred. Managers started appearing and asking to be involved in the project. It seems the two large concurrent IT projects were in trouble and experiencing large delays and cost overruns. By comparison SCRS was looking like a massive success. Cost overruns were tracking towards 20-30%, but covered by the client business unit which was effusively praising the final product. One manager offered to write us a piece in the company magazine. Not sure why he thought we couldn’t do that ourselves? Still the project manager went from being ignored, to fielding daily requests to help out or people offering “advice”. One positive is that when it came time for the team members to find new projects they all got their preferred choice.

Soon it was time for the end. We trained the support person. I flew up to Edinburgh to help deploy and troubleshoot the final installation. Unsurprisingly there were issues at this stage as we had not been allows to test this process before. I spent my two days in Scotland sitting in a office basement getting it all working fine. I can’t have seen more than a few minutes of sunshine during the entire visit. Then back via the RBS trough, aka the Edinburgh Airport British Airways business lounge (full of RBS employees grabbing as much food and alcohol as they could manage). When I got back I had a week of just waiting by myself (and the project manager) in case there were any problems. Everyone else had moved on. In fact the project manager had arranged for me to join another project too (more about that in Part 2), to ensure I was around just in case of any problems. In the end it wasn’t required, no one was ever called back to work on the SCRS. Instead during those last days we got fan mail. Seriously, at least three relationship managers sent us emails saying how much easier SCRS had made their jobs. It was amazing.

A year later, just after leaving RBS, I met the SCRS support guy at another person’s leaving do. I asked how it was going. He told me SCRS was the easiest system he had ever supported. In the more than a year since going live there had not been a single bug reported. The SCRS work was a few minutes every week setting up new users and running a backup. A system being used for a year, meeting all its customer’s requirements, without bugs. SCRS is probably the most successful project I have ever been involved with (apart from profitability, but that is a story for a later post). So that should be the end, right? Unfortunately not, there is one sad addendum.

Just after talking to the SCRS support person, I met and spoke to the SCRS project manager at the same leaving party. It turned out that on his performance review SCRS was classed as a failed project. I was aghast. I told him about the support person saying there were no bugs. He replied that didn’t matter. The project was overtime and budget. I pointed out that the client covered the budget overrun and the extra time was due to their extra requests. He said that the customer had no input to the project assessment process. Lastly, I pointed out that no other project in the building could be considered a success either by those measures. He agreed, but then told me that the two big concurrent projects were considered successes internally despite being relatively further over time and budget. The difference being that they both had more political clout behind them. Not all projects could be marked as successful, some had to fail according to the internal review system. IT projects were assessed on a curve relative to their peers. SCRS, with no important friends, was the designated failure. I told my ex-manager that his work and support was a core reason for SCRS’s obvious success by all reasonable measures – that didn’t make him any happier. Never have I been so relieved to have left an employer.

Click here for the story of my second project in RBS corporate & retail IT: RBS (First Time) – Part 2.


August 3, 2016

Glorious Day!

Tags:

Yes! This week is the culmination (but not termination) of the last four years’ work. My game Concealed Intent has reached 1.0 status and been fully released. No longer is it marked with the stigma of Early Access.

For the next week there is a 34% launch discount on Steam, and now for the first time it is also available on the Humble Store too (same discount).

Development work is not over, there will still at least be a version 1.1 update at some point. However, I think the largest part of game is now complete. Reaching a standard I felt happy about and then saying that people can pay for it without that Early Access caveat is a huge step. Also, as well as the 1.1 update, there will be work on another game (as stated in this year’s goals).


July 29, 2016

Payout

Tags:

Stranger things have happened, but not often.

After eight and a half years, I have actually earned real money from this blog. A few days ago Google paid me out just over £60 (the minimum) for the ads that have been displayed here since the beginning. For comparison, the hosting costs of this blog has been around £300 over the same time period, so financially this is a bad deal.

Another way to look at this is that there have been 333 posts here, so that is £0.18 per post – not a good salary, as most of them have taken a couple of hours. There have been a total of 216,000 visits since establishment, corresponding to £0.28 CPM, which is actually not too bad. So the problem is (relatively) few people read my posts.

As an update on my last state of the blog post, this blog is down to around 1500 visits per month. Which is a base earning of about £0.05/month, plus more from the occasional ad click. So the mathematically astute among you may work out that based on the previous post and the earnings rates I list above, it should still be many months before the £60 threshold was reached. Well done, you are right! However, I recently started a YouTube channel (and associated blog): A Gamedev Plays. There the CPM is closer to £0.70, plus clicks, and I’ve had a couple of clicks. Clearly, this is where ad-based content publishing is heading. The money from this is what pushed me over the edge.


June 27, 2016

A Gamedev Plays...

Tags: , , ,

I have set up a new website: A Gamedev Plays… will be dedicated to discussion and reviews of all sorts of games (but mostly games that I want to play). There is already an associated YouTube channel and Steam group. I will be using the Jarrah Technology Twitter and Facebook pages for promotional purposes at the moment.

I appear to be increasingly writing about games here. The desire to do so will only increase. Unsurprising, considering that most of my time is currently spent designing, building or just thinking about games. So I thought it best to split out the games posts into their own website, as such a monomanical focus was never the intention for this site. There may still be games posts here, but only ones I feel are particularly notable (similarly some of the old games posts from here may make it to the new site).

The plan for the new site is to play more games and write about them, both as a game design learning exercise and as motivation to create more myself. As a secondary goal, I also hope to create a small community around my games writing.

There is an explosion of game making at the moment. More and more games, many with innovative ideas are being released. I need to be aware of what is happening in game design. Since starting work on my own games I can’t help but try to analyse what makes a game work (or not!) when I play it. Being able to organise these thoughts into a coherent argument in a presentable form can only help me improve my future games.