When an application reaches production: DevOps, integrations and a secure cloud

ArcCloud / Engineer's workshop / 01

At 9:02 the customer paid. At 9:03 the application says they did not.

The provider accepted the payment, but the response did not arrive in time. The user refreshes the screen. The application retries the request. Moments later, two notifications arrive for the same transaction. In the logs, they look like several unrelated events. To the customer, this is simply one order whose status nobody can explain.

Problems like these rarely fit inside a single method. They span code, networks, databases, queues and deployment processes. You can have a well-written controller and still issue a document twice. Your pipeline can be green, yet after release you get a call saying employees cannot log in. You can run an application in the cloud and still have no way to restore its data.

This guide develops a way of thinking about the whole system. We connect every important decision to a symptom, a cause, a solution and a test capable of exposing the fault. That test will stay with you longer than the name of the next cloud service.

Read in order or return to a particular problem

The road to production

01 / Before there are services

First establish what the system promises people

Pracownia lets companies order parts. It has an Angular interface, a .NET API, a relational database and a background process. It connects to a payment provider and a warehouse. Once an order is approved, it sends confirmation and prepares a document. That is enough to encounter difficult problems. We do not need thirty microservices for that.

We start by talking to the person who handles orders. Does “accepted” mean saved, paid or reserved in the warehouse? Can the quantity change after payment starts? What do we do if payment succeeds but the goods are no longer available? Without these answers, a developer is not modeling a process. They are encoding their own guesses in if.

Separate states that change independently

A single Status field with values such as “new”, “paid”, “shipped” and “error” soon starts to lie. An order may be paid while still awaiting stock reservation. A payment may need investigation while its document already exists. We therefore model order, payment and fulfillment states separately. The customer screen combines these facts into a clear message: “Payment confirmed. Waiting for warehouse confirmation.”

Order:       Draft → Submitted → Cancelled
Payment:     Pending → Confirmed | Failed | NeedsReview
Fulfilment:  NotRequested → Reserving → Reserved | Rejected

Rule: create a sales document at most once
      for a particular confirmed transaction.
Rule: users from company A cannot read company B's orders.
Rule: a timeout does not automatically change Pending to Failed.

Transitions need explicit conditions. Payment confirmation should not move an order back to an earlier stage. A delayed “pending” message must not overwrite a confirmation already received. Where two people change the same record, concurrency control helps: an update with an expected version and a clear conflict, rather than silently overwriting someone else's work.

Choose simplicity you can maintain

Well-separated modules are enough to begin with: Orders, Payments and Fulfillment. The API handles interactions; a worker handles background tasks. Modules own their rules and do not change each other's data through convenient shortcuts. Splitting them into independent services makes sense when there is a concrete reason: separate teams, significantly different scale, required isolation or independent release cycles.

Every service split adds a network boundary, contract versioning and another place that can fail. Write a short architectural decision record: the problem, options considered, choice and condition for revisiting it. “We will separate report generation if it starts taking resources from the API despite limits” is more useful than a diagram that makes everything a microservice from day one.

02 / Modeling the boundary between systems

An integration starts with an agreement, not an HTTP client

An external warehouse calls a product an “item”, returns quantities as text and describes reservation errors using codes unfamiliar to the user. Do not spread that format throughout your domain. An integration adapter should translate the external contract into application concepts: reservation confirmed, out of stock, rejected data or an outcome that is not yet known.

Distinguish commands from events. ReserveStock is a request to perform work; it can be rejected. StockReserved describes a fact that has already happened. Precise names make it easier to decide who owns a decision and whether a message can be retried.

An example message needs fewer fields than a complex ORM object, but more than just an order identifier:

{
  "eventId": "evt-demo-0001",
  "type": "OrderSubmitted",
  "schemaVersion": 1,
  "occurredAt": "2026-10-13T09:00:00Z",
  "orderId": "ord-demo-0001",
  "tenantId": "tenant-demo-a",
  "correlationId": "corr-demo-0001"
}

eventId identifies a specific event, orderId a business object, and correlationId helps connect traces from one workflow. They are not interchangeable. The tenantId field does not grant permissions: the receiver must trust the producer and check the permitted context. In a public API, the company identifier comes from the user's authorized relationship, not from blindly accepted JSON.

A schema change is a deployment too

Start by adding optional fields and teaching consumers to ignore unknown ones. Changing an existing field's meaning breaks the contract even when the JSON still parses. A consumer should not turn an unknown currency into PLN by default. Maintain sample messages, producer and consumer contract tests, and a period when versions coexist. Store monetary values with an explicit currency and an agreed amount representation, without accidental floating-point arithmetic.

Not every response must arrive immediately. A long-running task can return 202 Accepted and an operation identifier whose status the client checks later. “Accepted for processing” must mean something durable. If a task exists only in process memory and disappears on restart, do not promise the user guaranteed completion.

Pracownia scenario / a failure you will not see locally

“I added retries. Now we sometimes create two invoices.”

In tests, the partner always responded. In our example, it saved the data but the connection broke before the response arrived. Our code repeated the command. The first attempt had not failed; its outcome was unknown. That distinction changes the entire recovery strategy.

03 / Uncertainty is part of the protocol

Retries only help once repetition has a defined meaning

Here, idempotency means that repeatedly performing the same logical operation does not create additional business effects. When creating an order, the client keeps the same key for retries. The server associates it with the user or company, the operation type and a fingerprint of the relevant data. Changing the quantity while reusing the same key must not quietly return the old order's result; it should produce an explicit conflict.

Check authentication and authorization before returning a stored result too. An old key must not bypass revoked permissions. Your database needs a uniqueness constraint and a transaction. Without that protection, “check whether it exists; if not, insert” loses to two concurrent requests. Also define what happens to operations in progress, how responses are stored and how long keys remain valid. Once a key expires, retry protection may disappear. A separate business constraint, such as a unique provider transaction identifier, still matters.

BEGIN
  claim (TenantId, Operation, IdempotencyKey)  -- UNIQUE
  verify the request fingerprint matches
  save Order
  save OutboxEvent with a durable EventId
  save the operation result associated with the key
COMMIT

Worker: read outbox → publish → mark sent
Consumer: BEGIN → claim (Consumer, EventId) UNIQUE
          → business change → COMMIT → ACK

Outbox closes one gap. It does not close every gap

Without an outbox, saving an order can succeed while sending its message to the queue fails. Saving the event alongside the order in the same transaction lets a worker return to it later. Another failure window remains: the broker accepts the message, but the worker dies before marking it sent. After restarting, it publishes the message again. The consumer therefore stores processed event identifiers and commits deduplication together with its own data changes.

Finish a duplicate of an already committed event without a new effect and acknowledge receipt. A conflict with an operation still in progress needs separately defined handling. If the effect is a call to another external service, the local transaction does not cover that server. You need idempotency at the partner too, your own reconciliation process or explicit compensation. A refund does not rewind time; it is a new operation that can also fail. Do not promise the team “exactly once” without stating the precise scope of that guarantee.

Set a retry budget before choosing the number of attempts

When reading partner data, consider bounded retries for transient failures, increasing delays with jitter and respecting Retry-After. A 400 response caused by an invalid product number will not fix itself a second later. A write timeout requires a different approach from a safe-read timeout. Every attempt consumes time, connections and the partner's allowance.

Suppose the user can wait no more than eight seconds. Three five-second attempts plus delays do not fit that promise. You need a deadline for the whole operation and cancellation propagated to dependencies. Do not independently add retries in three layers: four attempts at each of three levels can produce up to 64 calls to the deepest service. A circuit breaker temporarily cuts off an unreliable dependency; it does not replace idempotency or a concurrency limit.

Documentation: AWS: safe retries and the transactional outbox pattern.

04 / Public entry, limited trust

A webhook is a message from a partner. First check that it really came from them

Do not confirm a payment just because the browser returned to /success. JSON containing paid: trueis not enough either. Base the actual status on a verified provider message or an authorized call to its API, tied to the correct order and expected amount and currency.

In an HMAC signing scheme, verification usually requires the exact bytes of the request body. Parsing and reserializing JSON can change whitespace or field order, and therefore the signed material. Use the provider's official library and specification: how timestamps and content are combined, encoding and secret rotation are not universal.

A replay-protection time window must account for how that particular partner signs redeliveries. Rigidly rejecting every message older than five minutes without understanding the protocol can discard valid delayed events. Clock synchronization, deduplication and a manual reconciliation procedure are part of the solution.

Tests should cover a valid signature, an invalid signature, no signature, an oversized body, two concurrent copies of an event and reversed delivery order. Also test a worker crash after saving the effect but before acknowledging the message. That is often more valuable than another perfect JSON example.

One concrete protocol: GitHub: validating webhook signatures. Always check your own provider's parameters in its documentation.

05 / Architecture with trust boundaries

What can be public, and what should stay off the Internet?

“Private” restricts the network path. “Protected” also involves identity, permissions and data controls. These concepts do not replace each other. A public API can be properly protected, while an internal service available to too many accounts can be unsafe.

A signed URL is a confidential token: anyone holding it can use or forward it until it expires. Limit its scope and lifetime, and keep it out of public logs. Public frontend code is downloadable by definition. Minification or an environment variable embedded at build time will not hide an API key inside it. A public application identifier or API address may be visible; credentials that permit privileged operations must stay on the server or be replaced with an appropriate delegated-authorization mechanism.

In Azure, creating a Private Endpoint alone does not necessarily disable a service's existing public access. Verify the public network accesssetting, firewall rules and DNS. In AWS, likewise check routes, security groups, resource policies and endpoint policies. A test from inside the network does not prove that Internet access is blocked.

Documentation: Azure Private Endpoint and AWS IAM practices.

06 / Protection that also works outside the interface

A hidden button is not an authorization check

A customer from company A is logged in and can view their orders. They change the identifier in the URL to an order belonging to company B. If the server only checks token validity, it has just exposed someone else's data. Every object operation must check the user–company–resource–action relationship. A “not found” response can reduce disclosure of an object's existence, but the essential step is denying access before returning data.

tenant = authorizedMembership.For(currentUser)
order = Orders.SingleOrDefault(
    id == requestedId AND tenantId == tenant.Id)

if order is null: return NotFound
if !policy.CanCancel(currentUser, order): return Forbidden
order.Cancel(serverTime)

A token is not authentic simply because its JSON can be read. Use proven middleware and validate its signature, issuer, audience and lifetime. Privileged roles need a separate policy and MFA. Permission changes and session termination need defined behavior, especially when tokens remain valid for a long time.

Five places where security controls often fall apart

• Browser sessions. Choose cookies and tokens to suit the architecture. For cookie sessions, set Secure and HttpOnly, among other options, and deliberately choose SameSite behavior and CSRF protection. HttpOnly limits script access to a cookie, but does not solve XSS or CSRF by itself.
• CORS. It controls browser behavior when reading responses across origins. It does not replace authentication or stop arbitrary HTTP clients.
• User input. Parameterize SQL queries, encode data for its output context and avoid injecting untrusted HTML. Validate length, range and allowed values on the server.
• Attachments. Check size and actual type, generate your own filenames and keep files separate from application code. Archives can expand to enormous sizes. Scanning and quarantine must not create a direct path into a public directory.
• Fetching an external URL. An import or audit feature can be abused for SSRF. Restrict protocols and destinations, block internal and metadata addresses, and also check redirects and DNS resolution results. Add network-level egress controls.

Request limits must account for cost. A hundred simple reads and a hundred generated reports impose different loads. Limit expensive operations by authenticated identity and company; use network signals for public entry points too. Return a clear explanation when a limit is exceeded without exposing infrastructure details.

A WAF, dependency scanner and penetration test are useful, but they do not transfer responsibility for access rules from the application to a tool. Include an account from another company, an account without the required role, an expired session and a request sent without using the interface in your test plan. Nobody can honestly promise a system “immune to every intrusion”. You can consistently reduce the attack surface, detect abuse and practice revoking access.

For the test checklist: OWASP: object-level authorization and SSRF prevention.

07 / A repeatable process, not a hero's script

Good DevOps starts with being able to reproduce a deployment

Someone on the team “knows how to deploy it”. A configuration file lives on their computer, and several manual changes live on the server. While that person is available, everything seems efficient. Trouble arrives with a holiday, an incident or a second environment. Operational knowledge cannot exist only in someone's memory and terminal history.

The repository should describe the code, build definition, infrastructure and non-secret configuration. Secrets belong in an appropriate store; documentation identifies their names, owners and rotation procedure. Test and production environments need separate identities and access scopes. “A different database under the same administrator account” does not provide enough isolation.

Build once, promote the same result

If staging and production build the application separately, you tested a similar product, not necessarily the same one. Pin dependency resolution with a lock file, record the toolchain version and promote the same image, identified by its digest, between environments. The tag latest is not enough to tell you which code was running on Tuesday at 14:10.

Useful evidence includes the commit, pipeline run identifier, artifact checksum, test results, dependency inventory and release approval. An SBOM helps determine whether a discovered vulnerability affects your image. Signatures and provenance attestations help establish where the image came from. None of these documents alone means the code is secure.

CI/CD should not hold a permanent administrator key

OIDC federation lets a job exchange its identity for short-lived cloud permissions. Protection still depends on trust conditions: the correct issuer, token audience, repository, branch or protected environment. Simply setting id-token: write in GitHub Actions makes an OIDC token available; it does not grant unlimited access to AWS or Azure.

Check the actual format of the sub claim for your repository and provider. GitHub is changing how new repositories are identified; a sample policy copied from an old article may not match. On the cloud side, the role should permit deploying specific resources, not managing the entire account. A job testing an external pull request should not receive production credentials.

A green pipeline must prove something

A unit test checks a rule; an integration test should run a real database engine and demonstrate that transactions and constraints work. A contract test protects the boundary with a partner. An E2E test checks the key user journey. Introduce deliberate failures: access denial, a timeout, a duplicate and an incompatible schema. A hundred tests using a perfect mock do not establish correct transaction isolation.

Separately define what stops a release. An authorization or migration test failure should block deployment. Vulnerability scanning needs rules for assessing relevance, remediation deadlines and exceptions. A team that ignores a permanently red scanner has only added decoration to its pipeline.

Documentation: GitHub Actions and OIDC and secure workflow use.

08 / Old and new versions coexist

Rolling back an application does not automatically reverse data changes

The new version stores a different meaning in a field, a migration removes the old column, and minutes later the error rate rises. You restore the previous image. It still expects the old structure. The rollback button works technically, but the application no longer works for the business. A safe return must be planned before release.

Expand, migrate, then remove

With expand–contract, the first deployment adds a new structure while staying compatible with the old version. The next stage migrates data in batches and keeps writes compatible. The application then starts using the new model. Only after confirming that old processes and consumers no longer need the obsolete element does a separate change remove it.

1. Add new fields; the old version still reads the old ones.
2. Deploy compatible writes to both old and new formats.
3. Backfill historical data in small, resumable batches.
4. Compare completeness; switch reads under a feature flag.
5. Observe while retaining the option to switch reads back.
6. Remove the old format in a separate, later deployment.

Before switching, check every active write path, including old instances and workers. They must not quietly maintain only the old format. A backfill should check the record version or an appropriate update condition so it does not overwrite a newer change. Dual writes must be correct too. If both fields are in one database, use one transaction. If they are in two services, you need synchronization and reconciliation. A backfill must be safe to repeat; locking a table for a long time during business hours can cause more trouble than the schema change itself.

A gradual rollout needs an explicit stop condition

A new version can receive a small share of traffic before serving everyone. Watch errors, latency and business outcomes, not merely whether the process is alive. Define a threshold, measurement window and decision owner. A feature flag can disable a new path, but it will not reverse a financial operation already performed or replace data compatibility.

After release, run brief checks: a test-account login, an authorized read, a controlled operation and message processing by a worker. The environment and test data must be identifiable so monitoring does not confuse them with customer traffic. An automated production test must not accidentally charge a real card or send messages to random recipients.

Why compatibility is part of rollback: AWS Builders’ Library.

Pracownia scenario / on call at 14:17

“We added RAM. The problem came back an hour later.”

The importer reads the entire file, builds a list of all records and starts a task for each one. A larger machine only delayed the failure. We need to know how much work may be in progress, how large each item is and what happens when producers outrun consumers.

09 / Resources are part of the contract

We limit memory at several levels, not with one variable

Measure first. We care about the memory of the whole process or container, the GC-managed heap, allocation rate, GC pauses and the number of operations in flight. A heap that grows after successive full collections suggests a different issue from a stable heap with growing native memory. A one-off spike while generating a large file needs a different fix from an endlessly growing cache.

Typical causes are ordinary: ToListAsync() over a whole table, a huge JSON document in memory, a dictionary without expiry, undisposed streams, excessive parallel tasks or logging entire documents. Set maximum page, file, request and result sizes. Stream where the format allows it; read small batches and do not retain unnecessary copies.

A bounded queue is a deliberate response to overload

If 200 jobs arrive each second and you process 50, the backlog grows by roughly 150 every second. Without a limit, memory merely stores the problem. You can wait for capacity, reject work with an overload response or hand it durably to a broker. The choice depends on your agreement with the user. Silently dropping a job after displaying “accepted” is not a valid strategy.

var queue = Channel.CreateBounded<Job>(
    new BoundedChannelOptions(100)
    {
        FullMode = BoundedChannelFullMode.Wait,
        SingleReader = true,
        SingleWriter = false
    });

// Waits for capacity and observes cancellation.
await queue.Writer.WriteAsync(job, cancellationToken);

// Alternative: reject immediately if the channel is full.
bool accepted = queue.Writer.TryWrite(job);

An unlimited number of producers waiting in WriteAsync still keeps tasks and buffers in memory. Also limit concurrent requests or producers and waiting time. These last two lines show alternative ways to enqueue work: do not execute both for the same job. The channel lives in process memory and is not a durable broker. A capacity of 100 limits item count, not total size. A hundred objects each holding a 20 MB file still consume a lot of memory. It is better to pass a file identifier in private storage and read the file through a controlled stream.

A Kubernetes request is not a ceiling

requests.memory is used, among other things, when scheduling a pod onto a node. limits.memory defines a container limit enforced through operating-system mechanisms. A process may be terminated for running out of memory; the application will not always get a chance to handle an exception gracefully. A CPU limit can cause throttling, so an overly tight setting can increase latency and timeouts.

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1000m"
    memory: "512Mi"

This fragment does not describe a complete deployment. It covers one container. With multiple replicas, sidecars and node resources, total consumption differs. Choose values based on load testing, startup time and GC behavior; leave headroom, then observe whether the system needs it.

A .NET trap: “60” does not always mean sixty

A GC heap limit is not a limit on all process memory. Thread stacks, some runtime structures and native allocations remain outside it. Setting the heap to the entire container limit may leave too little room for everything else. First check runtime behavior inside the container; defaults account for environmental limits, and manual configuration should follow measurement.

# Environment variable: hexadecimal notation
DOTNET_GCHeapHardLimitPercent=0x3C

// runtimeconfig.json fragment: decimal number
"System.GC.HeapHardLimitPercent": 60

In an environment variable, the value 60 is interpreted as hexadecimal 0x60, or 96 in decimal. Check configuration precedence too: an explicit byte limit can cause the percentage to be ignored. Do not set every variant at once “just in case”.

Details: GC configuration in .NET, Kubernetes requests and limits and System.Threading.Channels.

10 / Every service brings cost and responsibility

AWS and Azure: choose the capabilities you need, not the biggest diagram

For a small team, a managed application environment can be a more sensible starting point than running its own Kubernetes cluster. A cluster provides control but adds upgrades, networking, policies, capacity and on-call responsibility. Choose it when the team needs those capabilities and can maintain them. Packaging an application in a container does not automatically require the orchestrator with the most options.

The following map shows examples of services with related purposes. They are not interchangeable one for one. Their semantics, limits, billing, regions and responsibility boundaries differ:

Domain boundaries and integration contracts are more portable than service names. An adapter can hide broker details, but it must not promise guarantees that one implementation cannot provide. If message order is a business requirement, define the scope of that ordering and verify it for the chosen configuration.

A budget alert does not stop the meter

Budgets help detect rising costs, but billing data arrives with a delay. Do not treat an alert at PLN 100 as a guarantee that the bill will never exceed that amount. Automated actions also have delays and limited scope. Stopping a database may interrupt the business while some charges continue.

Limit costs earlier: cap replicas, concurrent jobs, upload size, job duration, request volume and retention. Assign an owner and cost tags to resources. Check forgotten test environments, old snapshots, inter-region transfers, NAT and duplicated logs. Autoscaling without a defined maximum can turn a code defect into a financial problem.

Also define what happens when a limit is reached: reject new work, queue it with an estimated wait or restrict a particular feature. Never protect a cloud account solely with a counter in the browser. Final enforcement belongs on the server and platform.

The limits of budget mechanisms: Azure Cost Management and AWS Budgets.

11 / Measure the service from a person's perspective

The server returns 200. Why does the customer still have no report?

An API can be healthy while a worker has not collected a job for an hour. Alongside HTTP errors, you therefore need the age of the oldest message, the number of pending operations and end-to-end completion time. Average queue length does not tell you whether one customer is stuck forever.

For the Pracownia exercise, assume this goal: 99% of correctly accepted orders should receive warehouse confirmation within two minutes. This is an SLO for a specific capability, not a blanket promise for the whole application. Define where measurement starts and ends, exclusions and how partner failures count. Measurement should reflect the customer's experience even when the immediate cause is outside your server.

Propagate trace context through HTTP and messages using standard mechanisms such as OpenTelemetry. Add event and operation identifiers to logs. Do not log tokens, secrets, entire documents or unnecessary personal data. Give telemetry defined retention and access rules. Do not use orderId as a label on every metric: a huge number of unique values can increase costs and overload the monitoring system.

Three probes, three different questions

• Startup: has the application finished starting? This gives slower-starting processes time.
• Readiness: can this instance accept traffic now? Failure removes it from ready traffic destinations rather than automatically restarting it.
• Liveness: has the process become stuck in a way a restart could fix? Failure beyond a defined threshold can restart the container.

If liveness unconditionally depends on the database, a brief database outage can trigger a wave of application restarts. Design readiness deliberately too: if every instance depends on the same partner, removing them all may worsen the situation. Sometimes it is better to disable one capability and keep the others available.

A good alert says who should respond, what the impact is and where to look first. “CPU 85%” without context does not always justify waking someone. “No valid job has completed for five minutes, and the oldest has waited 12 minutes” points toward a specific diagnosis. Attach a short runbook: dashboard, first three checks, a safe action and an escalation path.

Documentation: Kubernetes probe semantics and OpenTelemetry in .NET.

12 / Calm starts before the alarm

The incident plan must work while the system's author is asleep

Return to the payment at the start of this article. We see “pending”, and the customer says they paid. The worst first response is to charge them again manually. First stop automatically creating further effects, identify the transactions and compare the provider's state with our records. Only then should you resume processing, flag the case for investigation or perform an agreed compensation.

A DLQ is not a bin nobody opens

A dead-letter queue holds messages that could not be processed under the defined rules. Each should have an identifiable cause, attempt count and owner. After a fix, do not blindly replay the entire queue at maximum speed: you may repeat the failure or overload the partner. Start with a small sample, monitor results and only then increase the rate.

A backup is the start of the evidence, not the end

Define RPO, the acceptable amount of data loss expressed in time, and RTO, the target time to restore service. An exercise RPO of 15 minutes and RTO of two hours are requirements to verify, not properties of a purchased machine. A daily backup will not meet the first goal without additional mechanisms.

Restore a backup in an isolated environment. Check encryption keys, permissions, file completeness and application-version compatibility. Verify business rules, record counts and document readability. Disable email sending and real payments in the recovery environment so the test does not repeat customer operations.

After restoring an older database, its state may lag behind events already sent to the partner. You need post-restore reconciliation. Protecting backups from deletion and separating backup permissions from the application's normal account reduces the risk that one compromised identity destroys both production and the last recovery option.

After an incident, record the timeline, impact, cause and concrete actions with owners. “Be more careful next time” is not a system fix. A duplicate test, adjusted limit, additional signal or easier rollback is. A good review helps the next person; it is not a search for someone to blame.

Backup verification: AWS Well-Architected: periodic recovery tests.

13 / Read it? Now try it yourself

.NET lab: break the flow before the network does

Download a small console project with source code and Polish instructions. It runs locally without a cloud account, keys or external NuGet packages. It covers eight self-checking scenarios: retry, payload conflict, a race between 32 requests, company-scoped keys, failed commit, publication failure, consumer deduplication and a full queue.

dotnet run --project ArcCloud.ProductionPatterns.Lab.csproj --configuration Release

The expected result is eight PASS messages and exit code 0. The project targets net8.0 for lab compatibility and requires the appropriate SDK reference packs. The README explains running on a newer runtime. For a new product, assess the currently supported .NET 10 LTS; this small exercise's target is not a version recommendation for a new system.

Three rounds that help you understand the mechanism

1. Predict the result. Before running the DispatcherCrash scenario, write down how many orders, deliveries and effects you expect. Only then read the result.
2. Introduce a controlled defect in a copy. Skip the inbox check. The duplicate test should detect an additional effect. If it still passes, check that you changed the right code and that the test measures the right thing.
3. Apply the principle to your own application. Find an operation that can repeat. Define its key, scope, retention period, transaction boundary and reconciliation process.

The package also includes worksheets: an integration contract, deployment plan and incident runbook. Fill them with real names, owners and thresholds. Replace an empty “monitoring: yes” with a specific dashboard address and the condition that prompts someone to act.

14 / “Done” needs evidence

How do you accept an application without mistaking completed tickets for quality?

Organize acceptance around scenarios rather than the number of completed tickets. Every promise needs an initial condition, an expected outcome and a place where the result is recorded. A new person can then repeat the check instead of taking someone's word that they “already clicked through it”.

Include the user's perspective in acceptance: does the message explain the delay, can the operation be resumed after a refresh, and can support see its state? Otherwise, a technically correct “202” can look like an empty page to the customer. This is where performance, availability and usability meet.

How to retain this knowledge

You do not need to memorize the whole article. Remember the questions that return in every project: “what if the response never arrives?”, “who can change this record?”, “what bounds the amount of work?”, “how will we restore the data?” and “which test could show that we are wrong?”

One article cannot replace practice, code review or responsibility for a running service. It can give you a repeatable method: assumption, implementation, deliberate failure, measurement and correction. That process remains useful even as library names change.

15 / Eight decisions you should be able to explain

A short quiz: what would you do on call?

Choose one answer per question. You will immediately see its consequences and can change your choice. This is a learning exercise, not a certificate. The result is neither sent to the server nor retained after a refresh.

Choose your answers. Your result stays in this browser tab.

Next steps / Turn knowledge into a concrete change

Your system does not need another generic list of best practices.

It needs answers about where it might lose data, repeat an operation, expose someone else's record or exceed infrastructure capacity. Then it needs prioritized fixes and tests that show whether a change works.

At ArcCloud, we examine applications, integrations and cloud environments. We can help take a problem from discovery to resolution: define the scope, demonstrate the risk with a concrete scenario and plan post-deployment verification. The right audit depends on what actually needs checking; a public website cannot reveal all the logic inside a private system.

Want more workshops like this? Explore ArcCloud Engineering Brief. You will find the current subscription offer below. This guide, quiz and lab remain available without a purchase.

16 / Go to the source when details matter

Documentation worth returning to

The links alongside each chapter lead to documentation from the technology authors and OWASP. The list below organizes further reading. Before using a configuration, check the runtime version, resource type and region. Service limits and capabilities change faster than fundamental design principles.

• Microsoft: Retry pattern — when retrying helps and when it makes a problem worse.
• AWS: idempotent APIs — operation identity and safe repetition.
• AWS: Transactional outbox — saving data and the intention to publish.
• GitHub: webhook signatures — exact content, signature and secret.
• OWASP API Security: BOLA — access to a specific object.
• OWASP: SSRF Prevention — boundaries for fetching external addresses.
• GitHub Actions: OIDC — trust and short-lived pipeline identity.
• Microsoft: GC configuration — heap limit formats and scope.
• Kubernetes: container resources — requests, limits and overload behavior.
• Kubernetes: probes — readiness, startup and process liveness.
• Microsoft: Azure for AWS professionals — comparing service roles without assuming full interchangeability.
• .NET support policy — choosing a supported version for a real product.

ArcCloud Studio educational material · checked 13 October 2026 · conceptual illustration and the fictional Pracownia system. More on code fundamentals: Clean code in C# and Clean Architecture.