Decide who owns each piece of data

For every object that moves between systems, decide which one is the source of truth, and in which direction changes flow. Two systems that both believe they own a customer record will disagree sooner or later. Writing the ownership down before building anything removes a whole category of later argument.

Authentication is part of the design

Integrations act as someone: a service account, or the user. That choice affects permissions, audit trails and what happens when a token expires or is revoked. OAuth2-connected user accounts, for example, need a plan for refresh failures and for users who disconnect.

Webhooks and polling

Webhooks are timely but unreliable on their own: deliveries can be missed, delayed, duplicated or arrive out of order. Polling is slower but easy to reason about. Many robust integrations use both: webhooks for speed, a periodic poll to catch what was missed.

Retries need idempotency

Retrying a failed call is only safe if repeating it cannot do harm. Give each operation a stable identifier so that sending it twice has the same effect as sending it once, and back off between attempts so a struggling system is not made worse. Decide when to stop retrying and where the failed item goes instead.

Expect duplicates and partial failure

Assume the same event can arrive more than once, and that a multi-step operation can succeed in the middle and fail at the end. Design each step so it can be repeated, and record enough state to resume.

Plan reconciliation

However careful the design, systems drift. A scheduled reconciliation that compares the two sides and reports (or repairs) differences turns silent corruption into a visible, fixable item.

Respect limits, and make failure visible

Rate limits, quotas and timeouts are part of the contract. Build in throttling, and make failures observable: structured logging, alerting, and a way for someone to see what is stuck and retry it by hand. A manual recovery path is not a sign of weakness; it is what makes the automated path trustworthy.

A checklist

• Source of truth defined for each object
• Authentication and token failure handled
• Webhooks backed by polling or reconciliation
• Idempotent operations with back-off
• A place for items that keep failing
• Monitoring and a manual recovery path

Start by separating the stack from the behaviour

Before choosing between an upgrade and a rewrite, write down two lists.

What is obsolete: the framework version, the Ruby runtime, the ORM or data layer, unsupported gems, the operating system, deployment steps that live in one person’s head.

What is valuable: the rules, edge cases and workflows that years of production use have taught the application. These are usually undocumented, and they are the part a rewrite is most likely to lose.

If the second list is long and the first is the real source of risk, an incremental path is usually the lower-risk answer.

When incremental modernisation is viable

Staged upgrades tend to work when:

• the application still runs and is understood well enough to be tested;
• the business logic is worth keeping;
• the obsolete layers can be replaced one at a time;
• the team can keep shipping while the work happens.

A programme that moved a Rails 2/Merb application to Rails 7.1 is an example of the shape this takes: framework, Ruby runtime (1.9 to 3.3), data layer (DataMapper to ActiveRecord) and operating system (Ubuntu 10 to 22.04) were each moved on their own terms rather than in one big-bang cutover. You can read the details in the case study.

Get test safety before you change anything

The most useful early work is often unglamorous: characterise how the application behaves today. Even a thin layer of tests around the workflows that matter lets you tell the difference between “the upgrade broke something” and “it always did that”. Without it, every later step is a leap of faith.

Sequence the layers deliberately

Order matters because the layers depend on each other.

1. Runtime and dependencies first, where possible. Move to a supported Ruby and replace abandoned gems so later steps are not blocked.
2. Framework in steps. Major Rails versions have documented upgrade paths; take them one version at a time rather than jumping.
3. Data layer when you can isolate it. Swapping an ORM touches every query. Do it behind tests, and ideally not in the same release as a framework change.
4. Infrastructure alongside, not after. Operating-system and deployment changes are coupled to the runtime, so plan them together.

Keep every step observable and reversible. If a step cannot be rolled back, make it smaller.

When a rewrite is justified

Sometimes it is. A rewrite makes sense when the business behaviour itself is changing, when the existing logic is not worth preserving, or when the cost of understanding the old system exceeds the cost of rebuilding it. The point is that this is a business-risk decision, not a preference for newer tools.

A quick test

Ask: if we replaced everything at once, what would we have no way to verify? If the answer is “most of what the business relies on”, staged modernisation is probably the safer route.

Keep deterministic code for what must be exact

Use ordinary code where correctness is binary or auditable:

• calculations and totals;
• permissions and access control;
• money movement;
• hard business rules;
• compliance logic where exactness is required.

These have right and wrong answers, and the system needs to give the same answer every time. Probabilistic behaviour here is a defect, not a feature.

Consider AI where meaning is ambiguous

AI earns its place where the input is language or the output is a judgement call:

• understanding free-text requests;
• semantic ranking and search;
• summarising long material;
• classifying messages or documents;
• recommending from many options;
• drafting text for a person to review.

Put a human in the loop where it matters

Assistance that a user can review is far safer than automation that acts unseen. Draft, suggest and rank; let a person approve anything with consequences. Design the review step as part of the feature rather than an afterthought, and decide in advance how you will check the output is good enough.

Search is often the practical starting point

Many business problems that sound like “we need AI” are really search problems. Combining conventional text matching with semantic (vector) search can help people find the right record quickly while the underlying data and rules remain deterministic. TickMessage, for example, uses trigram, full-text and vector search in PostgreSQL to find the right contact quickly — a search improvement, with the messaging logic itself remaining ordinary, testable code.

A decision test

For each workflow, ask:

1. Is there a single correct answer? If yes, use deterministic code.
2. Is the input ambiguous or linguistic? If yes, AI may help.
3. Can a person review the result before it matters? If not, be cautious.
4. How will we know it is working? If there is no answer, there is no feature yet.

The order is a promise

At checkout, the system promises something: these items, available, delivered by this time. Whether it can keep that promise depends on stock that may be perishable, suppliers with their own lead times, a fulfilment process, and a delivery route. If those live in separate tools joined by spreadsheets, the promise is only as good as the last person who updated them.

Where the gaps appear

Typical failure points are predictable: stock counts that lag reality, supplier orders raised by hand, delivery plans built from an export, and finance entering the same transaction twice. Each gap is small. Together they are the operation’s real cost.

Connect the lifecycle, not just the systems

The useful framing is a single order lifecycle that stock, purchasing, fulfilment, delivery, finance and customer service all read from and write to. Integrations then stop being add-ons and become how the operation expresses itself. An order changes state and the rest of the business can respond.

Design the boundaries

Not everything belongs in one platform. Accounting stays in the accounting system, route planning in the route planner. The engineering work is the boundary: what data moves, when, how exceptions are handled, and how both sides are reconciled.

Make exceptions visible

Out-of-stock substitutions, failed deliveries and supplier shortfalls are normal. A good system surfaces them to the people who can act, instead of hiding them until the customer calls.

This is the pattern behind our ecommerce and operations work and the ecommerce industry approach. If the difficult part of your system happens after checkout, talk to us about the workflow.

Why this matters:

If sophisticated polling operations with billion-dollar budgets can misread consumer sentiment so drastically across the US, UK, and Australia, it raises critical questions about the reliability of our own data-driven ecommerce systems.

Consider this: The last three US elections, two UK elections, and three Australian elections all saw significant forecasting failures. These aren’t just political mishaps – they’re warnings about the fragility of predictive systems that mirror our own ecommerce technology stacks.

Modern ecommerce operations share remarkable similarities with election campaign infrastructure:

• Both leverage recommendation engines for programmatic ad buying
• Both rely on A/B testing to optimize user journeys
• Both deploy sophisticated content personalization with behavioral triggers

The decision-making technology powering these systems isn’t just similar – it’s often identical. This parallel should give us pause.

What You Can Do:

As we approach the critical XMAS buying season, now is the perfect time to conduct a thorough review of your data-driven processes. Gather your team and suppliers for a comprehensive tech stack assessment. Remember: in ecommerce, faulty data doesn’t just impact your bottom line – it can be an extinction-level event for your business.

Appendix

1. The parallels between political polling and ecommerce analytics are striking. Both attempt to predict human behavior through data analysis, and both can fail spectacularly when their underlying assumptions are flawed.
2. For ecommerce businesses, the stakes are arguably higher than political polls. While a failed election forecast might embarrass pollsters, failed ecommerce predictions directly impact revenue and can lead to business failure.

Here’s an example of how to write a Procfile correctly for a Ruby on Rails application:

web: bundle exec rails server -p 3000
worker: bundle exec rake jobs:work

Let’s break down each part of this Procfile:

1. web: This line specifies the process type, in this case, it’s the web server process. When you run this line through a process management tool, it will start the web server for your Rails application.

2. bundle exec rails server: This is the command to run the web server process. It uses the bundle exec prefix to ensure that the Rails server is executed within the context of the application’s dependencies managed by Bundler.

3. worker: This line specifies another process type, in this case, it’s a worker process. Worker processes are often used for background jobs, and this line tells the process management tool to start the worker process.

4. bundle exec rake jobs:work: This is the command to run the worker process. It uses the bundle exec prefix just like the web server command to ensure that the worker process is executed within the application’s context.

When you have a Procfile set up with these lines, you can use process management tools like Foreman to start all the processes defined in the Procfile with a single command. This can be especially useful in development environments to mimic the environment in which the application will run in production.

The names of each process (e.g. web, worker) can be customised to suit your preferences, these titles are seen in your terminal output and as such are used to identify what processes are running or have failed.

Further, you can have a Procfile.dev in your application to run processes specific to local development environments. A new Rails application (version 7) with initial configuration set up to use Yarn/ESBuild/TailwindCSS would typically have a Procfile.dev that looks like this:

web: bin/rails server -p 3000
js: yarn build --watch
css: yarn build:css --watch

A separate script would be required to run this Procfile locally of which you can place in the bin folder. The standard practice is to have a script named dev in this folder which would contain the following:

#!/usr/bin/env sh

if ! gem list foreman -i --silent; then
  echo "Installing foreman..."
  gem install foreman
fi

exec foreman start -f Procfile.dev "$@"

Now when you run the bin/dev from the command line your processes will all start up using Foreman.

An alternative configuration for Rails developers is to optimise the operation of their apps using the Puma server. You can use the config file by using this line in the Procfile for reference:

web: bundle exec puma -C ./config/puma.rb

In summary, a Procfile in Ruby on Rails is a simple yet powerful way to manage multiple processes within your application. It allows you to define process types and their respective commands, making it easier to start and manage different parts of your application.

For any software engineer of note, your time is always at a premium, so the ability to quickly & efficiently navigate your way around the terminal will boost your productivity no end.

ctrl + A
#go to the START of the current line
ctrl + E
#go to the END of the current line
ctrl + L
#clear the screen
ctrl + U
#clear the line BEFORE the cursor position
ctrl + K
#clear the line AFTER the cursor position
#NB If you are at the end of the line the entire line is cleared
alt + F
#move forward one word position
alt + B
#move backwards one word position
tab
#autocomplete file & folder names...based on existing folder contents

The Big Picture:

Remember these venerable shortcuts?

CTRL-C for COPY
CTRL-X for CUT
CTRL-V for PASTE
CTRL-Z for UNDO

Of course you do, you use them daily.

Similarly, once you practice bash shortcuts they will speed your navigation & boost productivity x 100

Go Deeper…

Further Reading:

RedHatDotCom: 5 ways to navigate the Linux terminal faster

Why this matters:

One of the more complicated associations…..

ActiveRecord requires you to *define the relationships between two classes. Whilst the has_many and belongs_to association methods attract the most attention*, let’s consider the more obscure has_and_belongs_to_many.

Appendix

1. BTW Did anyone tell you that software development requires you to spend an insane amount of time thinking of the nature of things in the world and their interrelationships? All in the name of data modelling.

2. And let’s admit, especially for beginners, newbies for those rusty on ActiveRecord Associations, we’ll be tempted to try and jury-rig as many relationships into this association and make it work.

We eventually settled upon Hugo, the superfast framework built in Googles own language Go.

The motto of Hugo is build sites lightning fast & the build time doesn’t disappoint.

Deploying static sites are not at all straightforward & the options on the Hugo website don’t really offer anything that’s fast to setup & user friendly to maintain. Below is our guide to quickly deploying your Hugo site….

Assumptions & Prerequisites

1. You have Hugo installed
2. You’re ready to publish your content online.
3. You have initialized your root folder with git & have made a number of commitments.

Steps:

1. Ensure your baseURL is an empty string and not a url.

# config.yaml
title: "Pacifica Labs"
baseURL: ""  # note the absence of any url
languageCode: "en-AU"

2. Ensure Hugo builds all your content from scratch by removing your public folder:

rm -rf public/

3. Build your static site content…

$ hugo
Start building sites …

                   | EN
-------------------+-----
  Pages            | 21
  Paginator pages  |  0
  Non-page files   |  0
  Static files     | 18
  Processed images | 16
  Aliases          |  6
  Sitemaps         |  1
  Cleaned          |  0

Total in 263 ms

3. You will see newly compiled content in the public folder:

$ ls public/

404.html
about
articles
blog
docs
favicon.ico
fonts
gallery
index.html
(...)
(...)
(...)
(etc)

4. Create an empty repository on github and commit your entire project folder:

git add .

git commit -m "first commit"

5. Create a new branch called gh-pages which is the branch github pages will read from when deploying the static site:

6. Create a subtree

git push origin `git subtree split --prefix public master`:gh-pages --force

7. Wait til the pages notification changes from blue to green

8. Confirm your changes!

The Big Picture:

Key steps in deploying a Hugo repository require subtree usage.

Go Deeper…

(Further reading links here) This is a bash script I have written to automate the above steps of the deployment process.

Appendix

1. The top 4 global static site generators were Next.js, Hugo & Gatsby & Jekyll.
2. In benchmarking tests Hugo compiled 23x faster than Jekyll.