By Dmitry Baraishuk, chief technology officer and partner at custom software development company Belitsoft, which has an office in London

When your current or former vendor falls short of your expectations, produces subpar work, or is unable to complete the project at all, you might need software project rescue services.

The situation is urgent when you’re losing thousands of pounds and your customers are abandoning your product as they can’t finish their workflows because of ongoing problems and unplanned downtime.

But how do you tell a project that needs rescuing from a project that just needs one more sprint?

In my experience, there are five signs. To demonstrate them, I’ll use one of our recent rescue cases.

How it started

One of our clients is a digital media company with a 20-plus person team. They have built their own B2B database platform, a .NET web application, for over 15 years. The leads that come through the client’s websites are vital to the client’s business.

It was a Sunday, at the beginning of a month. Bots browsed the catalogue pages of the platform non-stop, and the platform started to go down. The site was off most of the month. Because the business depends on those leads, the company itself was threatened when the site went down.

Sign 1: You have already made several attempts – and none worked

This client was on Cloudflare previously. It kept the bots at the front door, but they went around it and reached the server directly, so the client moved to Akamai, which promised a more sophisticated solution. Akamai Security was installed and initially configured by a vendor that services the market as an Akamai partner. By the time the bots came, Akamai was in place and the vendor was managing it.

Some of what this vendor did about the bots didn’t help and some of it made things worse.

This is typical. Businesses may have to try three or more times to succeed because not every new vendor can deliver on what it promised. If you are keeping score of attempts instead of results, that is the first sign.

Sign 2: You are buying a box – but the problem is in your application

The client wanted a boxed solution for the bot problem. Everyone promises out-of-the-box protection with tools like these. Someone actually has to understand the various tools, write the rules and ensure the client’s application continues to work behind those rules.

Don’t think of buying a product as solving a problem. Both the application and a boxed solution like Akamai need to be properly configured and tuned, and without understanding the application, tuning can’t be done.

Sign 3: The advice is generic – and each step gets you no further, or makes it worse

The vendor was familiar with the Akamai product but didn’t understand the client’s application. So the advice was of the generic sort.

For example, Origin Health Detection, which was activated on that vendor’s recommendation, produced no noticeable result. Blocking individual IPs didn’t work either. The last move from the vendor was a TLS fingerprint rule. Real users then couldn’t get in any more than the bots could.

Tired of the finger-pointing (“it’s a PM mistake”, “no, it’s a dev mistake”, “no, it’s a new dev mistake”) and finding every part of the project like pulling teeth, our client decided to look for new technology partners.

Sign 4: You run a ‘big mechanical beast’ with no documentation

Another type of rescue work we do is that of ‘big mechanical beasts’ with dragged-down performance: software applications that have years of legacy code that no one else wants to maintain. That’s exactly what our client’s 15-year-old platform was.

The problem is that usually no documentation is left after the last team. Before fixing anything, we need to know what the application really consists of and where its critical parts are.

Sign 5: Nobody can describe what ‘normal’ looks like

Prior to protection rules and tuning, someone needs to understand how the application works with actual users. How many pages does a real person open in a minute? How frequently does a real user submit a form? What is the source of your audience?

The problem is that, as a business owner, you may not have asked these questions – and your provider may not have either. If no one can answer, the rules are being written blind. Without those answers, our client’s vendor could only offer generic advice – the last piece of which was a TLS fingerprint rule that kept real users out along with the bots.

This is why our client needed a development partner, not just Akamai consultancy and implementation services.

What rescue looks like in practice

How does it work? There are three steps.

First, stabilise, then understand the application. We learned enough about the system to make the first corrections to keep it going, but not enough to decide what to do with it. So Belitsoft began the audit with mapping the whole system. We spent time learning how the application works: which URLs exist and which don’t, how many pages a real person opens in a minute, how often a real person submits a form, where the client’s audience comes from. That understanding was the basis for the Akamai rules. Our engineer then went through the existing settings, noting what worked and what didn’t.

Second, tune the rules to the application – don’t rewrite the application. We never tell our clients that they have to start from scratch. Often some of the core logic is perfectly reusable. There’s no reason to waste developer resources rebuilding something that already works. In the project mentioned, geo-blocking of whole subnets did work, but the client also had real users in that region, so we turned it off and instead blocked the countries with the largest share of malicious traffic, reducing all traffic by 31%. The biggest win was rate limiting. For some reason, the vendor had resisted it, but Belitsoft insisted on implementing and fine-tuning it. Rate limiting rules blocked 10.4 million requests in one month. On top of that, Belitsoft added rules that come from an exact understanding of what the application does. Blocking query string parameters that the app never uses removed 1.1 million bot requests a month. A blacklist of paths that don’t exist, with a whitelist of the ones that do, removed another 0.7 million requests. Belitsoft also strengthened known-bot blocking and turned the WAF rules back on, and the client bought Akamai’s Content Protector module for bots that go through the site the way a real person would.

Third, cover the server and the code, not just the tool. The Akamai settings helped reduce the server load generated by bots. But the work was more than just setting up Akamai. On the server end, Belitsoft’s team increased the server capacity. The overload was removed by scaling the database and the web server vertically. On the application side, the team built a bot trap and optimised the catalogue pages most hit by bots.

The outcome

Prior to the month-long bot attack, the server was failing to serve just 3% of requests, which most users never noticed. After a month of bot scrapers going through the site, this figure went up to the point where pages were not opening at all. After Belitsoft’s work, it went down to 2% and below, which is actually better than before the attack.

The bots have not disappeared. They are still coming. In one recent month, Akamai received 31.8 million requests, 20.7 million of which – two-thirds – were blocked and didn’t reach the server. At a recent meeting the client said: “at the moment, fingers crossed, it’s not disrupting business, as before it was”.

To wrap up

This was a one-off rescue project and it is done, but Belitsoft is on standby to offer support. Bots are getting smarter every year, and what works today may not work so well over time. If you have projects that are crashing and burning, we can rescue them at the last minute, turn them around completely and get them across the finish line.